2024-09-16T14:54:23.960 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T14:54:23.976 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T14:54:24.062 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906794 branch: reef description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906794' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: reef ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 workunit: branch: reef sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1390 sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 sleep_before_teardown: 0 subset: 5319/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJiF00WUhcIkJ/1PTDrT+uxy9XFuLSU5hp0jKj8pXcSGVq2X8/u6cbMpva5FASZqc7nJviw/TC70O+V/zWzISRQ= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_22:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T14:54:24.062 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa; will attempt to use it 2024-09-16T14:54:24.063 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa/tasks 2024-09-16T14:54:24.063 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T14:54:24.065 INFO:teuthology.task.internal:Checking packages... 2024-09-16T14:54:24.087 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8' 2024-09-16T14:54:24.087 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T14:54:24.088 INFO:teuthology.packaging:ref: None 2024-09-16T14:54:24.088 INFO:teuthology.packaging:tag: None 2024-09-16T14:54:24.088 INFO:teuthology.packaging:branch: reef 2024-09-16T14:54:24.088 INFO:teuthology.packaging:sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T14:54:24.088 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=reef 2024-09-16T14:54:24.316 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-853-gae7f3733-1jammy 2024-09-16T14:54:24.318 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T14:54:24.325 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T14:54:24.325 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T14:54:24.342 INFO:teuthology.task.internal:Saving configuration 2024-09-16T14:54:24.351 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T14:54:24.358 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T14:54:24.385 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906794', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 14:47:46.989675', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOLtTbUyhipgzWzrpKsrdqcRtn5zW+XULUM+B5e1/DGI'} 2024-09-16T14:54:24.385 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T14:54:24.392 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-16T14:54:24.392 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T14:54:24.427 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff798ddf1c0>, signals=[15]) 2024-09-16T14:54:24.427 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T14:54:24.433 INFO:teuthology.task.internal:Opening connections... 2024-09-16T14:54:24.434 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T14:54:24.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:54:24.517 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T14:54:24.524 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-09-16T14:54:24.633 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-09-16T14:54:24.634 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-09-16T14:54:24.684 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T14:54:24.684 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="Ubuntu" 2024-09-16T14:54:24.684 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="22.04" 2024-09-16T14:54:24.684 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T14:54:24.684 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_CODENAME=jammy 2024-09-16T14:54:24.684 INFO:teuthology.orchestra.run.smithi039.stdout:ID=ubuntu 2024-09-16T14:54:24.684 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE=debian 2024-09-16T14:54:24.684 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T14:54:24.684 INFO:teuthology.orchestra.run.smithi039.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T14:54:24.685 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T14:54:24.685 INFO:teuthology.orchestra.run.smithi039.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T14:54:24.685 INFO:teuthology.orchestra.run.smithi039.stdout:UBUNTU_CODENAME=jammy 2024-09-16T14:54:24.685 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-09-16T14:54:24.708 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T14:54:24.716 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T14:54:24.722 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T14:54:24.722 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-09-16T14:54:24.732 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T14:54:24.738 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T14:54:24.738 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-09-16T14:54:24.788 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T14:54:24.821 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T14:54:24.837 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T14:54:24.837 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T14:54:24.837 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T14:54:24.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:54:24.842 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-16T14:54:24.888 INFO:teuthology.orchestra.run.smithi039.stdout:5.15.0-56-generic 2024-09-16T14:54:24.889 INFO:teuthology.task.kernel:Running kernel on smithi039: 5.15.0-56-generic 2024-09-16T14:54:24.889 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get clean 2024-09-16T14:54:25.021 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-16T14:54:25.177 INFO:teuthology.orchestra.run.smithi039.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T14:54:25.283 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T14:54:25.366 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T14:54:25.384 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T14:54:25.487 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T14:54:25.492 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T14:54:25.495 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T14:54:25.495 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T14:54:25.496 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T14:54:25.516 INFO:teuthology.orchestra.run.smithi039.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T14:54:25.520 INFO:teuthology.orchestra.run.smithi039.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T14:54:25.520 INFO:teuthology.orchestra.run.smithi039.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T14:54:25.525 INFO:teuthology.orchestra.run.smithi039.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T14:54:25.533 INFO:teuthology.orchestra.run.smithi039.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T14:54:25.535 INFO:teuthology.orchestra.run.smithi039.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T14:54:25.535 INFO:teuthology.orchestra.run.smithi039.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T14:54:25.535 INFO:teuthology.orchestra.run.smithi039.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T14:54:25.535 INFO:teuthology.orchestra.run.smithi039.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T14:54:25.546 INFO:teuthology.orchestra.run.smithi039.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T14:54:25.720 INFO:teuthology.orchestra.run.smithi039.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T14:54:25.859 INFO:teuthology.orchestra.run.smithi039.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,025 kB] 2024-09-16T14:54:26.177 INFO:teuthology.orchestra.run.smithi039.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T14:54:26.212 INFO:teuthology.orchestra.run.smithi039.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [353 kB] 2024-09-16T14:54:26.227 INFO:teuthology.orchestra.run.smithi039.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T14:54:26.228 INFO:teuthology.orchestra.run.smithi039.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T14:54:26.229 INFO:teuthology.orchestra.run.smithi039.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,441 kB] 2024-09-16T14:54:26.298 INFO:teuthology.orchestra.run.smithi039.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [420 kB] 2024-09-16T14:54:26.315 INFO:teuthology.orchestra.run.smithi039.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T14:54:26.315 INFO:teuthology.orchestra.run.smithi039.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T14:54:26.346 INFO:teuthology.orchestra.run.smithi039.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T14:54:26.366 INFO:teuthology.orchestra.run.smithi039.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T14:54:26.371 INFO:teuthology.orchestra.run.smithi039.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T14:54:26.372 INFO:teuthology.orchestra.run.smithi039.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T14:54:26.373 INFO:teuthology.orchestra.run.smithi039.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T14:54:26.374 INFO:teuthology.orchestra.run.smithi039.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T14:54:26.424 INFO:teuthology.orchestra.run.smithi039.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T14:54:26.424 INFO:teuthology.orchestra.run.smithi039.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T14:54:26.425 INFO:teuthology.orchestra.run.smithi039.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T14:54:26.457 INFO:teuthology.orchestra.run.smithi039.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T14:54:26.457 INFO:teuthology.orchestra.run.smithi039.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T14:54:26.458 INFO:teuthology.orchestra.run.smithi039.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T14:54:26.459 INFO:teuthology.orchestra.run.smithi039.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T14:54:26.459 INFO:teuthology.orchestra.run.smithi039.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T14:54:26.505 INFO:teuthology.orchestra.run.smithi039.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T14:54:32.244 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 16.0 MB in 2s (7,232 kB/s) 2024-09-16T14:54:33.597 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T14:54:33.615 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T14:54:33.682 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T14:54:33.897 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-16T14:54:33.897 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-16T14:54:34.093 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T14:54:34.093 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T14:54:34.093 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T14:54:34.093 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T14:54:34.093 INFO:teuthology.orchestra.run.smithi039.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T14:54:34.094 INFO:teuthology.orchestra.run.smithi039.stdout: python2.7-minimal 2024-09-16T14:54:34.094 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T14:54:34.095 INFO:teuthology.orchestra.run.smithi039.stdout:The following additional packages will be installed: 2024-09-16T14:54:34.095 INFO:teuthology.orchestra.run.smithi039.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T14:54:34.096 INFO:teuthology.orchestra.run.smithi039.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T14:54:34.096 INFO:teuthology.orchestra.run.smithi039.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T14:54:34.097 INFO:teuthology.orchestra.run.smithi039.stdout:Suggested packages: 2024-09-16T14:54:34.097 INFO:teuthology.orchestra.run.smithi039.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T14:54:34.097 INFO:teuthology.orchestra.run.smithi039.stdout:Recommended packages: 2024-09-16T14:54:34.097 INFO:teuthology.orchestra.run.smithi039.stdout: thermald 2024-09-16T14:54:34.165 INFO:teuthology.orchestra.run.smithi039.stdout:The following NEW packages will be installed: 2024-09-16T14:54:34.165 INFO:teuthology.orchestra.run.smithi039.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T14:54:34.165 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T14:54:34.165 INFO:teuthology.orchestra.run.smithi039.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T14:54:34.167 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages will be upgraded: 2024-09-16T14:54:34.167 INFO:teuthology.orchestra.run.smithi039.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T14:54:34.219 INFO:teuthology.orchestra.run.smithi039.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T14:54:34.219 INFO:teuthology.orchestra.run.smithi039.stdout:Need to get 113 MB of archives. 2024-09-16T14:54:34.219 INFO:teuthology.orchestra.run.smithi039.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T14:54:34.220 INFO:teuthology.orchestra.run.smithi039.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T14:54:34.685 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T14:54:34.800 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T14:54:35.635 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T14:54:35.635 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T14:54:35.635 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T14:54:35.774 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T14:54:35.797 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T14:54:36.212 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 113 MB in 2s (69.9 MB/s) 2024-09-16T14:54:36.391 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T14:54:38.340 INFO:teuthology.orchestra.run.smithi039.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T14:54:38.344 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:54:38.405 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:54:40.472 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T14:54:40.494 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:54:40.591 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:54:41.058 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T14:54:41.080 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:54:41.117 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:54:48.065 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T14:54:48.192 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T14:54:48.525 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T14:54:48.661 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T14:54:48.925 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T14:54:48.933 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T14:54:48.964 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T14:54:54.668 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T14:54:54.705 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:54:54.743 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:54:57.532 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T14:54:57.657 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T14:54:58.008 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T14:54:58.129 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:54:58.255 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T14:54:58.383 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:54:59.779 INFO:teuthology.orchestra.run.smithi039.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T14:54:59.780 INFO:teuthology.orchestra.run.smithi039.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T14:54:59.780 INFO:teuthology.orchestra.run.smithi039.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T14:54:59.780 INFO:teuthology.orchestra.run.smithi039.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T14:54:59.936 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:55:02.021 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:55:03.439 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T14:55:03.540 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T14:55:03.641 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:55:03.703 INFO:teuthology.orchestra.run.smithi039.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T14:55:03.703 INFO:teuthology.orchestra.run.smithi039.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T14:55:18.107 INFO:teuthology.orchestra.run.smithi039.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T14:55:18.107 INFO:teuthology.orchestra.run.smithi039.stdout:Sourcing file `/etc/default/grub' 2024-09-16T14:55:18.112 INFO:teuthology.orchestra.run.smithi039.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T14:55:18.144 INFO:teuthology.orchestra.run.smithi039.stdout:Generating grub configuration file ... 2024-09-16T14:55:18.451 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T14:55:18.470 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T14:55:18.757 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T14:55:18.763 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T14:55:18.835 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T14:55:18.841 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T14:55:18.997 INFO:teuthology.orchestra.run.smithi039.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T14:55:18.998 INFO:teuthology.orchestra.run.smithi039.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T14:55:18.998 INFO:teuthology.orchestra.run.smithi039.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T14:55:19.046 INFO:teuthology.orchestra.run.smithi039.stdout:done 2024-09-16T14:55:19.512 DEBUG:teuthology.orchestra.run.smithi039:> dpkg -s linux-image-generic 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi039.stdout:Package: linux-image-generic 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi039.stdout:Status: install ok installed 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi039.stdout:Priority: optional 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi039.stdout:Section: kernel 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi039.stdout:Installed-Size: 21 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi039.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: amd64 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi039.stdout:Source: linux-meta 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi039.stdout:Version: 5.15.0.121.121 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi039.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi039.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi039.stdout:Recommends: thermald 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi039.stdout:Description: Generic Linux kernel image 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: This package will always depend on the latest generic kernel image 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: available. 2024-09-16T14:55:19.536 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T14:55:19.536 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T14:55:19.536 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T14:55:19.536 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T14:55:19.536 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T14:55:19.536 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get clean 2024-09-16T14:55:19.715 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-16T14:55:19.836 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T14:55:19.845 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T14:55:19.862 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T14:55:19.956 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T14:55:21.318 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T14:55:21.336 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T14:55:21.401 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T14:55:21.616 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-16T14:55:21.616 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi039.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi039.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T14:55:21.861 INFO:teuthology.orchestra.run.smithi039.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T14:55:21.863 DEBUG:teuthology.orchestra.run.smithi039:> dpkg -s linux-image-generic 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Package: linux-image-generic 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Status: install ok installed 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Priority: optional 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Section: kernel 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Installed-Size: 21 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: amd64 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Source: linux-meta 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Version: 5.15.0.121.121 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T14:55:21.879 INFO:teuthology.orchestra.run.smithi039.stdout:Recommends: thermald 2024-09-16T14:55:21.880 INFO:teuthology.orchestra.run.smithi039.stdout:Description: Generic Linux kernel image 2024-09-16T14:55:21.880 INFO:teuthology.orchestra.run.smithi039.stdout: This package will always depend on the latest generic kernel image 2024-09-16T14:55:21.880 INFO:teuthology.orchestra.run.smithi039.stdout: available. 2024-09-16T14:55:21.881 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-16T14:55:21.928 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.3MSwjgPOTW 2024-09-16T14:55:21.928 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub/grub.cfg /tmp/tmp.3MSwjgPOTW 2024-09-16T14:55:21.984 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.3MSwjgPOTW 2024-09-16T14:55:22.100 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.3MSwjgPOTW is 10KB 2024-09-16T14:55:22.114 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.3MSwjgPOTW 2024-09-16T14:55:22.119 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T14:55:22.176 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T14:55:22.176 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T14:55:22.176 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T14:55:22.244 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T14:55:22.244 DEBUG:teuthology.orchestra.run.smithi039:> sudo update-grub 2024-09-16T14:55:23.566 INFO:teuthology.orchestra.run.smithi039.stderr:Sourcing file `/etc/default/grub' 2024-09-16T14:55:23.567 INFO:teuthology.orchestra.run.smithi039.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T14:55:23.573 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-09-16T14:55:23.748 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T14:55:23.754 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T14:55:23.979 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T14:55:23.984 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T14:55:24.026 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T14:55:24.031 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T14:55:24.157 INFO:teuthology.orchestra.run.smithi039.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T14:55:24.158 INFO:teuthology.orchestra.run.smithi039.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T14:55:24.158 INFO:teuthology.orchestra.run.smithi039.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T14:55:24.189 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-09-16T14:55:24.191 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-09-16T14:55:54.221 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T14:55:54.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T14:55:54.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:56:12.605 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-16T14:56:21.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T14:56:21.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:56:24.669 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-16T14:56:36.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T14:56:36.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:56:39.745 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-16T14:56:54.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T14:56:54.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:57:54.822 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T14:58:12.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T14:58:12.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:58:13.081 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-16T14:58:13.823 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T14:58:13.823 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T14:58:13.823 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-16T14:58:13.870 INFO:teuthology.orchestra.run.smithi039.stdout:5.15.0-121-generic 2024-09-16T14:58:13.870 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T14:58:13.870 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T14:58:13.870 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T14:58:14.871 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T14:58:14.871 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T14:58:14.942 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2024-09-16T14:58:14.975 DEBUG:teuthology.parallel:result is None 2024-09-16T14:58:14.976 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T14:58:14.984 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T14:58:14.984 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T14:58:14.990 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T14:58:14.997 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T14:58:15.008 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T14:58:15.009 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T14:58:15.044 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T14:58:15.053 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T14:58:15.053 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T14:58:15.106 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T14:58:15.114 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T14:58:15.116 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T14:58:15.122 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T14:58:15.122 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T14:58:15.175 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T14:58:15.184 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T14:58:15.185 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T14:58:15.222 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T14:58:15.270 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T14:58:15.318 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T14:58:15.318 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T14:58:15.382 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-09-16T14:58:15.457 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T14:58:15.464 INFO:teuthology.task.internal:Starting timer... 2024-09-16T14:58:15.464 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T14:58:15.588 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T14:58:15.661 DEBUG:teuthology.task.selinux:Excluding smithi039: OS 'ubuntu' does not support SELinux 2024-09-16T14:58:15.662 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T14:58:15.662 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T14:58:15.662 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T14:58:15.662 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T14:58:15.671 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T14:58:15.683 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T14:58:15.684 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T15:05:55.916 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com')] 2024-09-16T15:05:55.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T15:05:55.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T15:05:56.005 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-16T15:05:56.084 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T15:05:56.085 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T15:05:56.096 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T15:05:56.096 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T15:05:56.097 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T15:05:56.152 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T15:05:56.152 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: Command line: ntpd -gq 2024-09-16T15:05:56.152 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: ---------------------------------------------------- 2024-09-16T15:05:56.152 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T15:05:56.152 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T15:05:56.152 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: corporation. Support and training for ntp-4 are 2024-09-16T15:05:56.152 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: available at https://www.nwtime.org/support 2024-09-16T15:05:56.152 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: ---------------------------------------------------- 2024-09-16T15:05:56.153 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: proto: precision = 0.067 usec (-24) 2024-09-16T15:05:56.153 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: basedate set to 2022-02-04 2024-09-16T15:05:56.153 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: gps base set to 2022-02-06 (week 2196) 2024-09-16T15:05:56.153 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T15:05:56.154 INFO:teuthology.orchestra.run.smithi039.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T15:05:56.154 INFO:teuthology.orchestra.run.smithi039.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T15:05:56.155 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T15:05:56.155 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T15:05:56.155 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T15:05:56.155 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T15:05:56.155 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: Listen normally on 3 ens1f0 172.21.15.39:123 2024-09-16T15:05:56.155 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: Listen normally on 4 lo [::1]:123 2024-09-16T15:05:56.155 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1370%4]:123 2024-09-16T15:05:56.155 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:05:56 ntpd[18416]: Listening on routing socket on fd #22 for interface updates 2024-09-16T15:05:57.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:57 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:05:57.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:57 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:05:57.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:57 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:05:57.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:57 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:05:57.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:57 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:05:57.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:57 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:05:57.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:57 ntpd[18416]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:05:57.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:57 ntpd[18416]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:05:58.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:58 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:05:58.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:58 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:05:58.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:58 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:05:58.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:58 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:05:58.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:58 ntpd[18416]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:05:58.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:58 ntpd[18416]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:05:59.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:05:59.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:05:59.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:05:59.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:05:59.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:05:59.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:05:59.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:05:59.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:05:59.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:05:59.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:05:59.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:05:59.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:05:59 ntpd[18416]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:06:00.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:00 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:00.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:00 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:00.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:00 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:00.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:00 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:00.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:00 ntpd[18416]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:06:00.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:00 ntpd[18416]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:06:01.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:01.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:06:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:06:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:06:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:01 ntpd[18416]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:06:02.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:02 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:02.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:02 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:02.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:02 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:02.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:02 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:02.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:02 ntpd[18416]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:06:02.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:02 ntpd[18416]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:06:03.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:03.153 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:03.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:03.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:03.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:03.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:03.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:06:03.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:06:03.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:06:03.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:06:03.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:06:03.154 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:06:03.155 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 15:06:03 ntpd[18416]: ntpd: time slew -0.001037 s 2024-09-16T15:06:03.155 INFO:teuthology.orchestra.run.smithi039.stdout:ntpd: time slew -0.001037s 2024-09-16T15:06:03.155 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T15:06:03.155 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 15:06:03 ntpd[18416]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T15:06:03.205 INFO:teuthology.orchestra.run.smithi039.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T15:06:03.205 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================== 2024-09-16T15:06:03.205 INFO:teuthology.orchestra.run.smithi039.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:06:03.205 INFO:teuthology.orchestra.run.smithi039.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:06:03.205 INFO:teuthology.orchestra.run.smithi039.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:06:03.205 INFO:teuthology.orchestra.run.smithi039.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:06:03.206 INFO:teuthology.run_tasks:Running task install... 2024-09-16T15:06:03.214 DEBUG:teuthology.task.install:project ceph 2024-09-16T15:06:03.214 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}} 2024-09-16T15:06:03.214 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'} 2024-09-16T15:06:03.214 INFO:teuthology.task.install:Using flavor: default 2024-09-16T15:06:03.219 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T15:06:03.219 INFO:teuthology.task.install:extra packages: [] 2024-09-16T15:06:03.220 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-key list | grep Ceph 2024-09-16T15:06:03.308 INFO:teuthology.orchestra.run.smithi039.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T15:06:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T15:06:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T15:06:03.338 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T15:06:03.339 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T15:06:03.486 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default/ 2024-09-16T15:06:03.486 INFO:teuthology.task.install.deb:Package version is 18.2.4-845-ge4e227b3-1jammy 2024-09-16T15:06:03.596 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T15:06:03.597 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T15:06:03.611 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-16T15:06:03.789 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T15:06:03.799 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T15:06:03.817 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T15:06:03.841 INFO:teuthology.orchestra.run.smithi039.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T15:06:03.859 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T15:06:03.863 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T15:06:03.886 INFO:teuthology.orchestra.run.smithi039.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T15:06:03.909 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,411 B] 2024-09-16T15:06:03.939 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-09-16T15:06:04.157 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 28.8 kB in 0s (80.7 kB/s) 2024-09-16T15:06:05.197 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T15:06:05.214 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-845-ge4e227b3-1jammy cephadm=18.2.4-845-ge4e227b3-1jammy ceph-mds=18.2.4-845-ge4e227b3-1jammy ceph-mgr=18.2.4-845-ge4e227b3-1jammy ceph-common=18.2.4-845-ge4e227b3-1jammy ceph-fuse=18.2.4-845-ge4e227b3-1jammy ceph-test=18.2.4-845-ge4e227b3-1jammy ceph-volume=18.2.4-845-ge4e227b3-1jammy radosgw=18.2.4-845-ge4e227b3-1jammy python3-rados=18.2.4-845-ge4e227b3-1jammy python3-rgw=18.2.4-845-ge4e227b3-1jammy python3-cephfs=18.2.4-845-ge4e227b3-1jammy python3-rbd=18.2.4-845-ge4e227b3-1jammy libcephfs2=18.2.4-845-ge4e227b3-1jammy libcephfs-dev=18.2.4-845-ge4e227b3-1jammy librados2=18.2.4-845-ge4e227b3-1jammy librbd1=18.2.4-845-ge4e227b3-1jammy rbd-fuse=18.2.4-845-ge4e227b3-1jammy 2024-09-16T15:06:05.282 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T15:06:05.498 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-16T15:06:05.498 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-16T15:06:05.719 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T15:06:05.719 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T15:06:05.719 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T15:06:05.719 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T15:06:05.719 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T15:06:05.720 INFO:teuthology.orchestra.run.smithi039.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T15:06:05.720 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T15:06:05.720 INFO:teuthology.orchestra.run.smithi039.stdout:The following additional packages will be installed: 2024-09-16T15:06:05.720 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T15:06:05.721 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T15:06:05.721 INFO:teuthology.orchestra.run.smithi039.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-09-16T15:06:05.721 INFO:teuthology.orchestra.run.smithi039.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T15:06:05.722 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T15:06:05.723 INFO:teuthology.orchestra.run.smithi039.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T15:06:05.723 INFO:teuthology.orchestra.run.smithi039.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T15:06:05.723 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T15:06:05.723 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-09-16T15:06:05.723 INFO:teuthology.orchestra.run.smithi039.stdout:Suggested packages: 2024-09-16T15:06:05.723 INFO:teuthology.orchestra.run.smithi039.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-09-16T15:06:05.723 INFO:teuthology.orchestra.run.smithi039.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-09-16T15:06:05.723 INFO:teuthology.orchestra.run.smithi039.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-09-16T15:06:05.723 INFO:teuthology.orchestra.run.smithi039.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-09-16T15:06:05.723 INFO:teuthology.orchestra.run.smithi039.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-09-16T15:06:05.723 INFO:teuthology.orchestra.run.smithi039.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-09-16T15:06:05.724 INFO:teuthology.orchestra.run.smithi039.stdout: python3-watchdog 2024-09-16T15:06:05.724 INFO:teuthology.orchestra.run.smithi039.stdout:Recommended packages: 2024-09-16T15:06:05.724 INFO:teuthology.orchestra.run.smithi039.stdout: btrfs-tools 2024-09-16T15:06:05.831 INFO:teuthology.orchestra.run.smithi039.stdout:The following NEW packages will be installed: 2024-09-16T15:06:05.832 INFO:teuthology.orchestra.run.smithi039.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T15:06:05.832 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T15:06:05.832 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T15:06:05.832 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-09-16T15:06:05.832 INFO:teuthology.orchestra.run.smithi039.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T15:06:05.832 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-09-16T15:06:05.833 INFO:teuthology.orchestra.run.smithi039.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-09-16T15:06:05.833 INFO:teuthology.orchestra.run.smithi039.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T15:06:05.833 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T15:06:05.833 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-09-16T15:06:05.833 INFO:teuthology.orchestra.run.smithi039.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-09-16T15:06:05.833 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-16T15:06:05.834 INFO:teuthology.orchestra.run.smithi039.stdout: socat xmlstarlet 2024-09-16T15:06:05.977 INFO:teuthology.orchestra.run.smithi039.stdout:0 upgraded, 109 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T15:06:05.977 INFO:teuthology.orchestra.run.smithi039.stdout:Need to get 163 MB of archives. 2024-09-16T15:06:05.977 INFO:teuthology.orchestra.run.smithi039.stdout:After this operation, 746 MB of additional disk space will be used. 2024-09-16T15:06:05.977 INFO:teuthology.orchestra.run.smithi039.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.4-845-ge4e227b3-1jammy [3,451 kB] 2024-09-16T15:06:06.022 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-09-16T15:06:06.170 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.4-845-ge4e227b3-1jammy [3,108 kB] 2024-09-16T15:06:06.219 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.4-845-ge4e227b3-1jammy [683 kB] 2024-09-16T15:06:06.227 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.4-845-ge4e227b3-1jammy [357 kB] 2024-09-16T15:06:06.232 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.4-845-ge4e227b3-1jammy [32.8 kB] 2024-09-16T15:06:06.232 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.4-845-ge4e227b3-1jammy [180 kB] 2024-09-16T15:06:06.234 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.4-845-ge4e227b3-1jammy [63.7 kB] 2024-09-16T15:06:06.235 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.4-845-ge4e227b3-1jammy [333 kB] 2024-09-16T15:06:06.239 INFO:teuthology.orchestra.run.smithi039.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.4-845-ge4e227b3-1jammy [5,934 kB] 2024-09-16T15:06:06.308 INFO:teuthology.orchestra.run.smithi039.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T15:06:06.338 INFO:teuthology.orchestra.run.smithi039.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.4-845-ge4e227b3-1jammy [112 kB] 2024-09-16T15:06:06.339 INFO:teuthology.orchestra.run.smithi039.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.4-845-ge4e227b3-1jammy [445 kB] 2024-09-16T15:06:06.346 INFO:teuthology.orchestra.run.smithi039.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.4-845-ge4e227b3-1jammy [22.0 MB] 2024-09-16T15:06:06.485 INFO:teuthology.orchestra.run.smithi039.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T15:06:06.499 INFO:teuthology.orchestra.run.smithi039.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T15:06:06.574 INFO:teuthology.orchestra.run.smithi039.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T15:06:06.769 INFO:teuthology.orchestra.run.smithi039.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.4-845-ge4e227b3-1jammy [5,005 kB] 2024-09-16T15:06:06.822 INFO:teuthology.orchestra.run.smithi039.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T15:06:06.826 INFO:teuthology.orchestra.run.smithi039.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.4-845-ge4e227b3-1jammy [241 kB] 2024-09-16T15:06:06.828 INFO:teuthology.orchestra.run.smithi039.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T15:06:06.829 INFO:teuthology.orchestra.run.smithi039.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.4-845-ge4e227b3-1jammy [130 kB] 2024-09-16T15:06:06.830 INFO:teuthology.orchestra.run.smithi039.stdout:Get:23 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.4-845-ge4e227b3-1jammy [1,494 kB] 2024-09-16T15:06:06.843 INFO:teuthology.orchestra.run.smithi039.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.4-845-ge4e227b3-1jammy [6,085 kB] 2024-09-16T15:06:06.846 INFO:teuthology.orchestra.run.smithi039.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T15:06:06.853 INFO:teuthology.orchestra.run.smithi039.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T15:06:06.854 INFO:teuthology.orchestra.run.smithi039.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T15:06:06.855 INFO:teuthology.orchestra.run.smithi039.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T15:06:06.870 INFO:teuthology.orchestra.run.smithi039.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T15:06:06.871 INFO:teuthology.orchestra.run.smithi039.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T15:06:06.871 INFO:teuthology.orchestra.run.smithi039.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T15:06:06.912 INFO:teuthology.orchestra.run.smithi039.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T15:06:06.938 INFO:teuthology.orchestra.run.smithi039.stdout:Get:33 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.4-845-ge4e227b3-1jammy [22.4 MB] 2024-09-16T15:06:06.956 INFO:teuthology.orchestra.run.smithi039.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T15:06:06.957 INFO:teuthology.orchestra.run.smithi039.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T15:06:06.957 INFO:teuthology.orchestra.run.smithi039.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T15:06:06.958 INFO:teuthology.orchestra.run.smithi039.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T15:06:06.958 INFO:teuthology.orchestra.run.smithi039.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T15:06:06.959 INFO:teuthology.orchestra.run.smithi039.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T15:06:06.964 INFO:teuthology.orchestra.run.smithi039.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T15:06:06.997 INFO:teuthology.orchestra.run.smithi039.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T15:06:06.998 INFO:teuthology.orchestra.run.smithi039.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T15:06:07.082 INFO:teuthology.orchestra.run.smithi039.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T15:06:07.084 INFO:teuthology.orchestra.run.smithi039.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T15:06:07.085 INFO:teuthology.orchestra.run.smithi039.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T15:06:07.086 INFO:teuthology.orchestra.run.smithi039.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T15:06:07.088 INFO:teuthology.orchestra.run.smithi039.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T15:06:07.090 INFO:teuthology.orchestra.run.smithi039.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T15:06:07.091 INFO:teuthology.orchestra.run.smithi039.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T15:06:07.168 INFO:teuthology.orchestra.run.smithi039.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T15:06:07.168 INFO:teuthology.orchestra.run.smithi039.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T15:06:07.169 INFO:teuthology.orchestra.run.smithi039.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T15:06:07.252 INFO:teuthology.orchestra.run.smithi039.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T15:06:07.254 INFO:teuthology.orchestra.run.smithi039.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T15:06:07.258 INFO:teuthology.orchestra.run.smithi039.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T15:06:07.260 INFO:teuthology.orchestra.run.smithi039.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T15:06:07.262 INFO:teuthology.orchestra.run.smithi039.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T15:06:07.264 INFO:teuthology.orchestra.run.smithi039.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T15:06:07.269 INFO:teuthology.orchestra.run.smithi039.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T15:06:07.335 INFO:teuthology.orchestra.run.smithi039.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T15:06:07.390 INFO:teuthology.orchestra.run.smithi039.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T15:06:07.391 INFO:teuthology.orchestra.run.smithi039.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.4-845-ge4e227b3-1jammy [14.1 kB] 2024-09-16T15:06:07.392 INFO:teuthology.orchestra.run.smithi039.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.4-845-ge4e227b3-1jammy [834 kB] 2024-09-16T15:06:07.395 INFO:teuthology.orchestra.run.smithi039.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T15:06:07.402 INFO:teuthology.orchestra.run.smithi039.stdout:Get:65 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.4-845-ge4e227b3-1jammy [2,159 kB] 2024-09-16T15:06:07.428 INFO:teuthology.orchestra.run.smithi039.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.4-845-ge4e227b3-1jammy [240 kB] 2024-09-16T15:06:07.431 INFO:teuthology.orchestra.run.smithi039.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.4-845-ge4e227b3-1jammy [145 kB] 2024-09-16T15:06:07.433 INFO:teuthology.orchestra.run.smithi039.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.4-845-ge4e227b3-1jammy [2,233 kB] 2024-09-16T15:06:07.457 INFO:teuthology.orchestra.run.smithi039.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.4-845-ge4e227b3-1jammy [8,625 kB] 2024-09-16T15:06:07.536 INFO:teuthology.orchestra.run.smithi039.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T15:06:07.537 INFO:teuthology.orchestra.run.smithi039.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T15:06:07.540 INFO:teuthology.orchestra.run.smithi039.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T15:06:07.540 INFO:teuthology.orchestra.run.smithi039.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T15:06:07.544 INFO:teuthology.orchestra.run.smithi039.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T15:06:07.544 INFO:teuthology.orchestra.run.smithi039.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T15:06:07.544 INFO:teuthology.orchestra.run.smithi039.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T15:06:07.547 INFO:teuthology.orchestra.run.smithi039.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T15:06:07.548 INFO:teuthology.orchestra.run.smithi039.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T15:06:07.581 INFO:teuthology.orchestra.run.smithi039.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.4-845-ge4e227b3-1jammy [14.3 kB] 2024-09-16T15:06:07.582 INFO:teuthology.orchestra.run.smithi039.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.4-845-ge4e227b3-1jammy [42.8 MB] 2024-09-16T15:06:07.621 INFO:teuthology.orchestra.run.smithi039.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T15:06:07.662 INFO:teuthology.orchestra.run.smithi039.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T15:06:07.666 INFO:teuthology.orchestra.run.smithi039.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T15:06:07.667 INFO:teuthology.orchestra.run.smithi039.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T15:06:07.724 INFO:teuthology.orchestra.run.smithi039.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T15:06:07.724 INFO:teuthology.orchestra.run.smithi039.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T15:06:07.725 INFO:teuthology.orchestra.run.smithi039.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T15:06:07.726 INFO:teuthology.orchestra.run.smithi039.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T15:06:07.727 INFO:teuthology.orchestra.run.smithi039.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T15:06:07.728 INFO:teuthology.orchestra.run.smithi039.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T15:06:07.784 INFO:teuthology.orchestra.run.smithi039.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T15:06:07.785 INFO:teuthology.orchestra.run.smithi039.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T15:06:07.792 INFO:teuthology.orchestra.run.smithi039.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T15:06:07.866 INFO:teuthology.orchestra.run.smithi039.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T15:06:07.870 INFO:teuthology.orchestra.run.smithi039.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T15:06:07.871 INFO:teuthology.orchestra.run.smithi039.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T15:06:07.880 INFO:teuthology.orchestra.run.smithi039.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T15:06:07.887 INFO:teuthology.orchestra.run.smithi039.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T15:06:07.897 INFO:teuthology.orchestra.run.smithi039.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T15:06:07.905 INFO:teuthology.orchestra.run.smithi039.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T15:06:07.948 INFO:teuthology.orchestra.run.smithi039.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T15:06:07.952 INFO:teuthology.orchestra.run.smithi039.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T15:06:07.968 INFO:teuthology.orchestra.run.smithi039.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T15:06:08.029 INFO:teuthology.orchestra.run.smithi039.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T15:06:08.031 INFO:teuthology.orchestra.run.smithi039.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T15:06:08.468 INFO:teuthology.orchestra.run.smithi039.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.4-845-ge4e227b3-1jammy [122 kB] 2024-09-16T15:06:08.470 INFO:teuthology.orchestra.run.smithi039.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.4-845-ge4e227b3-1jammy [38.3 kB] 2024-09-16T15:06:08.471 INFO:teuthology.orchestra.run.smithi039.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.4-845-ge4e227b3-1jammy [11.7 MB] 2024-09-16T15:06:08.627 INFO:teuthology.orchestra.run.smithi039.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.4-845-ge4e227b3-1jammy [93.1 kB] 2024-09-16T15:06:09.199 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 163 MB in 3s (58.4 MB/s) 2024-09-16T15:06:09.336 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libfmt8:amd64. 2024-09-16T15:06:09.428 INFO:teuthology.orchestra.run.smithi039.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T15:06:09.434 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-09-16T15:06:09.519 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-16T15:06:09.780 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T15:06:09.796 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T15:06:09.830 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T15:06:10.158 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T15:06:10.185 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T15:06:10.225 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T15:06:10.561 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T15:06:10.588 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T15:06:10.628 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T15:06:11.015 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T15:06:11.042 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T15:06:11.081 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:06:11.518 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T15:06:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T15:06:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:06:11.921 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T15:06:11.947 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T15:06:11.995 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:06:12.324 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T15:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T15:06:12.382 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T15:06:12.685 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librados2. 2024-09-16T15:06:12.712 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../008-librados2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librados2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:13.196 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librbd1. 2024-09-16T15:06:13.223 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../009-librbd1_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:13.263 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librbd1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:13.707 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T15:06:13.734 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../010-libcephfs2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:13.773 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libcephfs2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:14.026 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rados. 2024-09-16T15:06:14.042 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../011-python3-rados_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:14.076 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rados (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:14.396 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T15:06:14.422 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T15:06:14.462 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-ceph-argparse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:14.714 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T15:06:14.730 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../013-python3-cephfs_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:14.772 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cephfs (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:15.101 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T15:06:15.127 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T15:06:15.167 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:15.470 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T15:06:15.497 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T15:06:15.528 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T15:06:15.906 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T15:06:15.933 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T15:06:15.972 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T15:06:16.217 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T15:06:16.233 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../017-python3-rbd_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:16.275 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rbd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:16.628 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T15:06:16.655 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T15:06:16.694 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T15:06:16.981 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librgw2. 2024-09-16T15:06:17.007 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../019-librgw2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:17.039 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librgw2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:17.533 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T15:06:17.549 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:17.591 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:17.911 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T15:06:17.938 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T15:06:17.977 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T15:06:18.289 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T15:06:18.315 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:18.355 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libradosstriper1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:18.717 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-common. 2024-09-16T15:06:18.744 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../023-ceph-common_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:18.775 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:20.064 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-base. 2024-09-16T15:06:20.091 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../024-ceph-base_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:20.140 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-base (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:20.752 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T15:06:20.778 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T15:06:20.818 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T15:06:21.096 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T15:06:21.122 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T15:06:21.154 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T15:06:21.432 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T15:06:21.448 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T15:06:21.482 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T15:06:21.777 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T15:06:21.803 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T15:06:21.843 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T15:06:22.146 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T15:06:22.172 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T15:06:22.212 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T15:06:22.490 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T15:06:22.517 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T15:06:22.556 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T15:06:22.801 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-portend. 2024-09-16T15:06:22.817 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-09-16T15:06:22.851 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T15:06:23.154 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T15:06:23.181 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T15:06:23.220 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T15:06:23.573 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T15:06:23.600 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T15:06:23.640 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T15:06:23.985 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T15:06:24.012 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T15:06:24.052 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T15:06:24.355 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T15:06:24.382 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T15:06:24.421 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T15:06:24.741 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T15:06:24.768 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T15:06:24.816 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T15:06:25.119 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-mako. 2024-09-16T15:06:25.146 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T15:06:25.185 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T15:06:25.522 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T15:06:25.549 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T15:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T15:06:25.891 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T15:06:25.918 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T15:06:25.951 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T15:06:26.246 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-webob. 2024-09-16T15:06:26.273 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T15:06:26.313 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T15:06:26.666 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T15:06:26.693 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T15:06:26.736 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T15:06:27.061 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T15:06:27.087 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T15:06:27.127 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T15:06:27.455 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-paste. 2024-09-16T15:06:27.482 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T15:06:27.530 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T15:06:27.867 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T15:06:27.882 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T15:06:27.966 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T15:06:28.278 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T15:06:28.305 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T15:06:28.345 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T15:06:28.606 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T15:06:28.633 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T15:06:28.664 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T15:06:28.992 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T15:06:29.019 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T15:06:29.059 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T15:06:29.414 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T15:06:29.441 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T15:06:29.480 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T15:06:30.017 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-idna. 2024-09-16T15:06:30.044 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../049-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T15:06:30.083 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T15:06:30.437 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T15:06:30.463 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T15:06:30.503 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T15:06:30.873 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-requests. 2024-09-16T15:06:30.900 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T15:06:30.940 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T15:06:31.251 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T15:06:31.278 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T15:06:31.318 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T15:06:31.604 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T15:06:31.620 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T15:06:31.663 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:32.035 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T15:06:32.061 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:32.092 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:32.404 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T15:06:32.431 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../055-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T15:06:32.471 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T15:06:32.774 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T15:06:32.800 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../056-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T15:06:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T15:06:33.143 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T15:06:33.170 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../057-ceph-mgr_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:33.210 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:33.580 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T15:06:33.607 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../058-ceph-mon_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:33.646 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mon (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:34.267 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T15:06:34.294 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../059-ceph-osd_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:34.333 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-osd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:35.387 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph. 2024-09-16T15:06:35.403 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../060-ceph_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:35.437 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:35.723 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T15:06:35.750 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../061-ceph-fuse_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:35.790 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:36.160 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T15:06:36.187 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../062-ceph-mds_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:36.243 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mds (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:36.597 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package cephadm. 2024-09-16T15:06:36.612 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../063-cephadm_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:36.646 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:36.941 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T15:06:36.968 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../064-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T15:06:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T15:06:37.286 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../065-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T15:06:37.361 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T15:06:38.425 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-babel. 2024-09-16T15:06:38.452 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../066-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T15:06:38.491 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T15:06:38.828 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T15:06:38.855 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../067-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T15:06:38.894 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T15:06:39.990 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T15:06:40.018 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../068-ceph-mgr-cephadm_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T15:06:40.056 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:40.419 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T15:06:40.446 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T15:06:40.485 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T15:06:40.755 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-py. 2024-09-16T15:06:40.783 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-09-16T15:06:40.821 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T15:06:41.134 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T15:06:41.161 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T15:06:41.200 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T15:06:41.528 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-toml. 2024-09-16T15:06:41.555 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-09-16T15:06:41.594 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T15:06:41.906 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T15:06:41.933 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T15:06:41.989 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T15:06:42.335 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T15:06:42.362 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T15:06:42.401 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T15:06:42.688 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-routes. 2024-09-16T15:06:42.704 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T15:06:42.737 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T15:06:43.083 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T15:06:43.110 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T15:06:43.149 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:44.372 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T15:06:44.400 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T15:06:44.430 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T15:06:44.917 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T15:06:44.934 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T15:06:44.975 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T15:06:45.354 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T15:06:45.381 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T15:06:45.420 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T15:06:45.732 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T15:06:45.759 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T15:06:45.798 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T15:06:46.549 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T15:06:46.577 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T15:06:46.615 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:47.519 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T15:06:47.546 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T15:06:47.585 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T15:06:47.957 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T15:06:47.984 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T15:06:48.023 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T15:06:48.343 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T15:06:48.371 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T15:06:48.401 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T15:06:48.680 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T15:06:48.696 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-09-16T15:06:48.738 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T15:06:49.058 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T15:06:49.086 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T15:06:49.125 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T15:06:49.445 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T15:06:49.473 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T15:06:49.520 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T15:06:49.831 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T15:06:49.859 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T15:06:49.898 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T15:06:50.202 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T15:06:50.229 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T15:06:50.267 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T15:06:50.965 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T15:06:50.982 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T15:06:51.023 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:51.318 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T15:06:51.346 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T15:06:51.385 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T15:06:51.730 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T15:06:51.757 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T15:06:51.796 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T15:06:52.066 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package jq. 2024-09-16T15:06:52.082 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T15:06:52.115 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T15:06:52.352 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package socat. 2024-09-16T15:06:52.368 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T15:06:52.401 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T15:06:52.713 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T15:06:52.730 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T15:06:52.771 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T15:06:53.126 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-test. 2024-09-16T15:06:53.142 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../096-ceph-test_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:53.184 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-test (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:55.172 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T15:06:55.200 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../097-ceph-volume_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T15:06:55.238 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-volume (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:55.802 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T15:06:55.830 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:55.876 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libcephfs-dev (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:06:56.188 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T15:06:56.215 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T15:06:56.246 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T15:06:56.551 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T15:06:56.609 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T15:06:57.029 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T15:06:57.057 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T15:06:57.104 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T15:06:57.374 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T15:06:57.402 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T15:06:57.432 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T15:06:57.744 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T15:06:57.771 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T15:06:57.810 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T15:06:58.256 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T15:06:58.284 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T15:06:58.322 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T15:06:58.634 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T15:06:58.662 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T15:06:58.700 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T15:06:58.970 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T15:06:58.998 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T15:06:59.036 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T15:06:59.749 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package radosgw. 2024-09-16T15:06:59.777 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../107-radosgw_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:06:59.815 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking radosgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:00.543 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T15:07:00.557 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../108-rbd-fuse_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T15:07:00.593 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking rbd-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:01.273 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T15:07:01.532 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T15:07:01.658 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T15:07:01.925 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T15:07:02.281 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T15:07:02.722 INFO:teuthology.orchestra.run.smithi039.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T15:07:02.785 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:03.319 INFO:teuthology.orchestra.run.smithi039.stdout:Adding system user cephadm....done 2024-09-16T15:07:03.449 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T15:07:03.759 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T15:07:04.035 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T15:07:04.161 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T15:07:04.413 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T15:07:04.672 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T15:07:04.798 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T15:07:05.117 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T15:07:05.486 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T15:07:05.753 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T15:07:06.015 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T15:07:06.266 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T15:07:06.534 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T15:07:06.793 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-ceph-argparse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:07.110 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T15:07:07.303 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T15:07:07.546 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T15:07:07.646 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T15:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T15:07:08.174 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T15:07:08.450 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T15:07:09.019 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T15:07:09.679 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T15:07:09.913 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T15:07:10.048 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T15:07:10.174 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T15:07:10.483 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T15:07:10.901 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T15:07:11.152 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T15:07:11.537 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T15:07:11.821 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T15:07:12.206 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T15:07:12.465 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T15:07:12.591 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:12.893 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T15:07:13.244 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T15:07:14.399 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:07:14.533 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T15:07:14.801 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T15:07:14.927 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T15:07:15.045 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T15:07:15.321 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T15:07:15.572 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T15:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:07:15.966 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-16T15:07:16.092 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T15:07:16.410 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T15:07:16.719 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T15:07:17.020 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T15:07:17.279 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T15:07:17.539 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T15:07:17.807 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T15:07:17.933 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T15:07:18.225 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T15:07:18.351 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T15:07:18.652 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T15:07:18.929 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T15:07:19.171 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T15:07:19.574 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T15:07:19.832 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T15:07:19.958 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T15:07:20.352 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T15:07:20.602 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T15:07:20.862 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:07:20.988 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T15:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T15:07:21.632 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T15:07:21.908 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T15:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T15:07:22.310 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T15:07:22.577 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T15:07:22.837 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T15:07:23.213 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T15:07:23.506 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T15:07:23.866 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T15:07:23.992 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librados2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:24.118 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librgw2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:24.244 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:24.354 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libcephfs2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:24.454 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libradosstriper1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:24.674 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T15:07:24.951 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librbd1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:25.077 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:25.203 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:25.413 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T15:07:25.414 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T15:07:25.963 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libcephfs-dev (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:26.097 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rados (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:26.255 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T15:07:27.521 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rbd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:27.656 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up rbd-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:27.782 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:27.907 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cephfs (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:28.075 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:28.568 INFO:teuthology.orchestra.run.smithi039.stdout:Adding group ceph....done 2024-09-16T15:07:29.045 INFO:teuthology.orchestra.run.smithi039.stdout:Adding system user ceph....done 2024-09-16T15:07:29.205 INFO:teuthology.orchestra.run.smithi039.stdout:Setting system user ceph properties....done 2024-09-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi039.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T15:07:29.368 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T15:07:29.674 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T15:07:30.185 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-test (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:30.311 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up radosgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:30.818 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T15:07:30.819 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T15:07:31.287 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-base (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:31.658 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T15:07:32.145 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mds (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:32.381 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T15:07:32.381 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:33.104 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T15:07:33.104 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T15:07:33.592 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-osd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:33.852 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T15:07:33.852 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T15:07:34.327 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:34.445 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:34.596 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mon (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:34.811 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T15:07:34.811 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T15:07:35.264 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:35.407 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:35.532 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:35.683 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-volume (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T15:07:36.153 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T15:07:37.388 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T15:07:37.887 INFO:teuthology.orchestra.run.smithi039.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T15:07:37.893 DEBUG:teuthology.parallel:result is None 2024-09-16T15:07:37.893 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T15:07:38.033 DEBUG:teuthology.orchestra.run.smithi039:> dpkg-query -W -f '${Version}' ceph 2024-09-16T15:07:38.052 INFO:teuthology.orchestra.run.smithi039.stdout:18.2.4-845-ge4e227b3-1jammy 2024-09-16T15:07:38.052 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845-ge4e227b3-1jammy 2024-09-16T15:07:38.052 INFO:teuthology.task.install:The correct ceph version 18.2.4-845-ge4e227b3-1jammy is installed. 2024-09-16T15:07:38.054 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T15:07:38.054 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T15:07:38.054 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T15:07:38.108 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T15:07:38.108 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T15:07:38.108 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T15:07:38.163 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T15:07:38.218 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T15:07:38.218 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T15:07:38.218 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T15:07:38.275 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T15:07:38.330 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T15:07:38.330 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T15:07:38.330 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T15:07:38.387 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T15:07:38.442 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T15:07:38.459 INFO:tasks.workunit:Pulling workunits from ref ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T15:07:38.459 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T15:07:38.459 INFO:tasks.workunit:timeout=3h 2024-09-16T15:07:38.460 INFO:tasks.workunit:cleanup=True 2024-09-16T15:07:38.460 DEBUG:teuthology.orchestra.run.smithi039:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:07:38.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:07:38.491 INFO:teuthology.orchestra.run.smithi039.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T15:07:38.491 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:07:38.539 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T15:07:38.539 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T15:07:38.587 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T15:07:38.637 INFO:tasks.workunit.client.0.smithi039.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T15:08:33.865 INFO:tasks.workunit.client.0.smithi039.stderr:Updating files: 92% (11914/12888) Updating files: 93% (11986/12888) Updating files: 94% (12115/12888) Updating files: 95% (12244/12888) Updating files: 96% (12373/12888) Updating files: 97% (12502/12888) Updating files: 98% (12631/12888) Updating files: 99% (12760/12888) Updating files: 100% (12888/12888) Updating files: 100% (12888/12888), done. 2024-09-16T15:08:34.905 INFO:tasks.workunit.client.0.smithi039.stderr:Note: switching to 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'. 2024-09-16T15:08:34.905 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T15:08:34.905 INFO:tasks.workunit.client.0.smithi039.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr: git switch -c 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr:Or undo this operation with: 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr: git switch - 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T15:08:34.906 INFO:tasks.workunit.client.0.smithi039.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T15:08:34.907 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T15:08:34.907 INFO:tasks.workunit.client.0.smithi039.stderr:HEAD is now at ae7f37330c1 Merge pull request #59795 from zdover23/wip-doc-2024-09-15-backport-59613-to-reef 2024-09-16T15:08:34.914 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T15:08:34.964 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T15:08:34.964 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T15:08:35.010 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-09-16T15:08:35.011 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-09-16T15:08:35.011 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-09-16T15:08:35.088 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T15:08:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T15:08:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:08:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:08:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T15:08:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T15:08:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T15:08:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T15:08:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T15:08:35.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T15:08:35.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/bad-inc-map 2024-09-16T15:08:35.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-09-16T15:08:35.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-09-16T15:08:35.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:37595 2024-09-16T15:08:35.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:37595 2024-09-16T15:08:35.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-09-16T15:08:35.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-09-16T15:08:35.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=d2a8b793-b226-47f2-a458-7e430807d9d0 --auth-supported=none ' 2024-09-16T15:08:35.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:37595 ' 2024-09-16T15:08:35.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-09-16T15:08:35.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T15:08:35.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-09-16T15:08:35.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-09-16T15:08:35.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-09-16T15:08:35.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-09-16T15:08:35.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2024-09-16T15:08:35.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2024-09-16T15:08:35.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-16T15:08:35.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:08:35.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-16T15:08:35.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:08:35.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:08:35.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:08:35.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:08:35.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:08:35.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:08:35.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:08:35.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:08:35.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:08:35.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:08:35.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:08:35.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:08:35.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:08:35.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:08:35.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:08:35.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:08:35.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:08:35.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:08:35.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-16T15:08:35.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:08:35.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:08:35.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21279 2024-09-16T15:08:35.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21279 2024-09-16T15:08:35.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:08:35.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:08:35.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2024-09-16T15:08:35.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:08:35.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:08:35.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21279 2024-09-16T15:08:35.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21279 2024-09-16T15:08:35.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:08:35.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T15:08:35.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T15:08:35.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:08:35.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-09-16T15:08:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-09-16T15:08:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-09-16T15:08:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-09-16T15:08:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2024-09-16T15:08:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:08:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:08:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:08:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2024-09-16T15:08:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-09-16T15:08:35.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:08:35.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:08:35.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:08:35.653 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:08:35.653 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:08:35.653 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21279 2024-09-16T15:08:35.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21279/$cluster-$name.asok' 2024-09-16T15:08:35.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21279/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:08:35.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:08:35.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:08:35.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:08:35.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:08:35.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:08:35.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:08:35.855 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:08:35.855 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:08:35.855 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:08:35.856 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:08:35.856 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:08:35.856 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21279 2024-09-16T15:08:35.856 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21279/ceph-mon.a.asok 2024-09-16T15:08:35.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:08:35.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21279/ceph-mon.a.asok config get fsid 2024-09-16T15:08:35.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:08:35.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:08:35.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:08:35.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:08:35.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:08:35.976 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:08:35.976 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:08:35.976 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:08:35.976 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:08:35.976 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:08:35.976 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21279 2024-09-16T15:08:35.976 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21279/ceph-mon.a.asok 2024-09-16T15:08:35.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:08:35.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21279/ceph-mon.a.asok config get mon_host 2024-09-16T15:08:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-09-16T15:08:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2024-09-16T15:08:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:08:36.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:08:36.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:08:36.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2024-09-16T15:08:36.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:08:37.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:08:37.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:08:37.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:08:37.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:08:37.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:08:37.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21279 2024-09-16T15:08:37.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21279/$cluster-$name.asok' 2024-09-16T15:08:37.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:08:37.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21279/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:08:37.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-09-16T15:08:37.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d2a8b793-b226-47f2-a458-7e430807d9d0 --auth-supported=none --mon-host=127.0.0.1:37595 ' 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:08:37.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21279 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21279/$cluster-$name.asok' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21279/$cluster-$name.asok' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:08:37.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:08:37.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:08:37.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:08:37.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:08:37.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2024-09-16T15:08:37.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:08:37.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf730696-dd39-4ea4-b84b-4c7aa370fdf2 2024-09-16T15:08:37.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bf730696-dd39-4ea4-b84b-4c7aa370fdf2' 2024-09-16T15:08:37.157 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 bf730696-dd39-4ea4-b84b-4c7aa370fdf2 2024-09-16T15:08:37.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:08:37.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1SehmZdF2ChAAoAgqgMgc1bppdQ+txU623g== 2024-09-16T15:08:37.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1SehmZdF2ChAAoAgqgMgc1bppdQ+txU623g=="}' 2024-09-16T15:08:37.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf730696-dd39-4ea4-b84b-4c7aa370fdf2 -i td/bad-inc-map/0/new.json 2024-09-16T15:08:37.552 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:08:37.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2024-09-16T15:08:37.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d2a8b793-b226-47f2-a458-7e430807d9d0 --auth-supported=none --mon-host=127.0.0.1:37595 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1SehmZdF2ChAAoAgqgMgc1bppdQ+txU623g== --osd-uuid bf730696-dd39-4ea4-b84b-4c7aa370fdf2 2024-09-16T15:08:37.621 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:37.618+0000 7f14ea3657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:08:37.643 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:37.642+0000 7f14ea3657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:08:37.646 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:37.646+0000 7f14ea3657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:08:37.647 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:37.646+0000 7f14ea3657c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-09-16T15:08:37.647 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:37.646+0000 7f14ea3657c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-09-16T15:08:37.647 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:37.646+0000 7f14ea3657c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-09-16T15:08:38.043 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:38.042+0000 7f14ea3657c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-09-16T15:08:43.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-09-16T15:08:43.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:08:43.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:08:43.849 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:08:43.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:08:44.095 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:08:44.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:08:44.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:08:44.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d2a8b793-b226-47f2-a458-7e430807d9d0 --auth-supported=none --mon-host=127.0.0.1:37595 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:08:44.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:08:44.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:08:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:44.126+0000 7f93c7b807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:08:44.131 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:44.130+0000 7f93c7b807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:08:44.132 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:44.130+0000 7f93c7b807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:08:44.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:08:44.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:08:44.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:08:44.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:08:44.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:08:44.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:08:44.316 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:08:44.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:08:44.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:08:44.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:08:44.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:08:44.688 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:44.686+0000 7f93c7b807c0 -1 Falling back to public interface 2024-09-16T15:08:45.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:08:45.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:08:45.530 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:08:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:08:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:08:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:08:45.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:08:46.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:08:46.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:08:46.831 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:08:46.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:08:46.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:08:46.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:08:47.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:08:47.280 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:08:47.278+0000 7f93c7b807c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:08:48.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:08:48.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:08:48.259 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:08:48.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:08:48.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:08:48.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:08:48.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:08:49.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:08:49.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:08:49.799 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:08:49.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:08:49.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:08:49.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:08:50.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:08:51.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:08:51.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:08:51.215 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:08:51.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:08:51.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:08:51.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:08:51.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:08:52.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:08:52.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:08:52.652 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:08:52.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:08:52.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:08:52.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:08:53.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:08:54.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:08:54.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:08:54.222 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:08:54.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:08:54.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:08:54.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:08:54.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:08:55.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:08:55.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:08:55.671 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:08:55.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:08:55.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:08:55.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:08:56.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:08:57.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:08:57.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:08:57.212 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:08:57.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:08:57.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:08:57.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:08:57.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:08:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:08:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:08:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:08:58.654 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:08:58.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:08:58.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:08:59.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:00.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:00.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:00.248 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:09:00.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:09:00.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:00.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:09:00.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:01.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:01.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:01.672 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:09:01.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:09:01.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:01.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:09:02.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:03.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:03.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:03.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:09:03.260 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:09:03.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:03.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:09:03.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:04.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:04.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:04.901 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:09:04.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:09:04.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:04.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:09:05.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:06.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:06.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:06.731 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:09:06.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:09:06.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:06.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:09:07.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:08.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:08.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:08.526 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:09:08.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:09:08.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:08.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:09:09.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:10.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:10.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:10.506 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:09:10.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:09:10.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:10.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:09:11.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:12.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:12.235 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T15:09:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T15:09:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:12.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:09:12.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:13.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:13.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:13.990 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-09-16T15:09:13.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T15:09:13.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:13.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:09:14.428 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3246432608,v1:127.0.0.1:6801/3246432608] [v2:127.0.0.1:6802/3246432608,v1:127.0.0.1:6803/3246432608] exists,up bf730696-dd39-4ea4-b84b-4c7aa370fdf2 2024-09-16T15:09:14.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:09:14.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d2a8b793-b226-47f2-a458-7e430807d9d0 --auth-supported=none --mon-host=127.0.0.1:37595 ' 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-09-16T15:09:14.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21279 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21279/$cluster-$name.asok' 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21279/$cluster-$name.asok' 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:09:14.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:09:14.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:09:14.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-16T15:09:14.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-16T15:09:14.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:09:14.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:09:14.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:09:14.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:09:14.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:09:14.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:09:14.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2024-09-16T15:09:14.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:09:14.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=017bbcbb-6378-4adf-999e-386589c35e62 2024-09-16T15:09:14.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 017bbcbb-6378-4adf-999e-386589c35e62' 2024-09-16T15:09:14.434 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 017bbcbb-6378-4adf-999e-386589c35e62 2024-09-16T15:09:14.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:09:14.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaSuhmikHAGhAAbCf3ulBavorG5tiN5477nw== 2024-09-16T15:09:14.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaSuhmikHAGhAAbCf3ulBavorG5tiN5477nw=="}' 2024-09-16T15:09:14.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 017bbcbb-6378-4adf-999e-386589c35e62 -i td/bad-inc-map/1/new.json 2024-09-16T15:09:14.962 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:09:14.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2024-09-16T15:09:14.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d2a8b793-b226-47f2-a458-7e430807d9d0 --auth-supported=none --mon-host=127.0.0.1:37595 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAaSuhmikHAGhAAbCf3ulBavorG5tiN5477nw== --osd-uuid 017bbcbb-6378-4adf-999e-386589c35e62 2024-09-16T15:09:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:15.010+0000 7f251e1777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:15.013 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:15.010+0000 7f251e1777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:15.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:15.014+0000 7f251e1777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:15.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:15.014+0000 7f251e1777c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-09-16T15:09:15.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:15.014+0000 7f251e1777c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-09-16T15:09:15.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:15.014+0000 7f251e1777c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-09-16T15:09:15.094 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:15.094+0000 7f251e1777c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-09-16T15:09:21.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-09-16T15:09:21.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:09:21.042 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:09:21.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:09:21.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:09:21.598 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:09:21.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:09:21.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d2a8b793-b226-47f2-a458-7e430807d9d0 --auth-supported=none --mon-host=127.0.0.1:37595 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:09:21.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:09:21.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:09:21.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:09:21.629 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:21.626+0000 7f81f97137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:21.650 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:21.650+0000 7f81f97137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:21.651 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:21.650+0000 7f81f97137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:22.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:09:22.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:09:22.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:09:22.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:09:22.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:09:22.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:22.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:09:22.039 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:09:22.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:22.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:22.212 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:22.210+0000 7f81f97137c0 -1 Falling back to public interface 2024-09-16T15:09:22.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:23.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:23.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:23.498 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:09:23.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:09:23.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:23.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:24.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:24.860 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:24.858+0000 7f81f97137c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:09:25.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:25.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:25.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:09:25.061 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:09:25.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:25.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:25.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:26.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:26.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:26.596 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:09:26.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:09:26.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:26.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:27.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:28.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:28.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:28.101 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:09:28.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:09:28.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:28.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:28.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:29.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:29.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:29.700 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:09:29.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:09:29.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:29.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:30.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:31.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:31.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:31.130 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:09:31.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:09:31.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:31.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:31.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:32.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:32.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:32.662 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:09:32.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:09:32.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:32.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:33.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:34.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:34.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:34.155 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:09:34.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:09:34.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:34.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:34.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:35.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:35.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:35.651 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:09:35.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:09:35.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:35.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:36.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:37.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:37.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:37.183 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:09:37.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:09:37.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:37.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:37.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:38.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:38.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:38.612 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:09:38.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:09:38.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:38.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:39.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:40.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:40.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:40.369 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:09:40.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:09:40.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:40.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:41.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:42.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:42.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:42.246 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:09:42.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:09:42.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:42.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:43.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:44.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:44.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:44.222 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:09:44.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:09:44.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:44.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:45.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:46.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:46.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:46.062 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:09:46.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:09:46.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:46.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:46.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:47.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:47.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:47.816 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:09:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:09:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:47.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:48.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:49.405 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:09:49.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:09:49.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:49.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:09:49.866 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2333741688,v1:127.0.0.1:6811/2333741688] [v2:127.0.0.1:6812/2333741688,v1:127.0.0.1:6813/2333741688] exists,up 017bbcbb-6378-4adf-999e-386589c35e62 2024-09-16T15:09:49.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:09:49.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:09:49.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d2a8b793-b226-47f2-a458-7e430807d9d0 --auth-supported=none --mon-host=127.0.0.1:37595 ' 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-09-16T15:09:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-09-16T15:09:49.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:09:49.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:09:49.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-16T15:09:49.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:09:49.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:09:49.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:09:49.869 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:09:49.869 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:09:49.869 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21279 2024-09-16T15:09:49.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21279/$cluster-$name.asok' 2024-09-16T15:09:49.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21279/$cluster-$name.asok' 2024-09-16T15:09:49.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:09:49.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:09:49.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:09:49.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-16T15:09:49.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-16T15:09:49.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:09:49.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:09:49.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:09:49.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:09:49.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:09:49.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:09:49.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2024-09-16T15:09:49.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:09:49.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a36fbba2-bb79-4917-9677-18ee52bc8b44 2024-09-16T15:09:49.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a36fbba2-bb79-4917-9677-18ee52bc8b44' 2024-09-16T15:09:49.873 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 a36fbba2-bb79-4917-9677-18ee52bc8b44 2024-09-16T15:09:49.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:09:49.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9SuhmdLHzNBAAM5X6RVu+N6jyfORycQ7HnQ== 2024-09-16T15:09:49.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9SuhmdLHzNBAAM5X6RVu+N6jyfORycQ7HnQ=="}' 2024-09-16T15:09:49.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a36fbba2-bb79-4917-9677-18ee52bc8b44 -i td/bad-inc-map/2/new.json 2024-09-16T15:09:50.412 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:09:50.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2024-09-16T15:09:50.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d2a8b793-b226-47f2-a458-7e430807d9d0 --auth-supported=none --mon-host=127.0.0.1:37595 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9SuhmdLHzNBAAM5X6RVu+N6jyfORycQ7HnQ== --osd-uuid a36fbba2-bb79-4917-9677-18ee52bc8b44 2024-09-16T15:09:50.463 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:50.460+0000 7f93b1dc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:50.466 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:50.460+0000 7f93b1dc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:50.467 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:50.464+0000 7f93b1dc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:50.467 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:50.464+0000 7f93b1dc87c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-09-16T15:09:50.467 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:50.464+0000 7f93b1dc87c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-09-16T15:09:50.467 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:50.464+0000 7f93b1dc87c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-09-16T15:09:50.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:50.548+0000 7f93b1dc87c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-09-16T15:09:56.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-09-16T15:09:56.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:09:56.670 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T15:09:56.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T15:09:56.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:09:57.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T15:09:57.227 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:09:57.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d2a8b793-b226-47f2-a458-7e430807d9d0 --auth-supported=none --mon-host=127.0.0.1:37595 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:09:57.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:09:57.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:09:57.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:09:57.257 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:57.252+0000 7f41cba347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:57.273 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:57.268+0000 7f41cba347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:57.275 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:57.272+0000 7f41cba347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:09:57.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T15:09:57.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:09:57.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:09:57.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:09:57.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:09:57.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:57.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:09:57.651 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:09:57.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:57.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:09:57.844 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:09:57.840+0000 7f41cba347c0 -1 Falling back to public interface 2024-09-16T15:09:58.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:09:59.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:09:59.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:09:59.081 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:09:59.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:09:59.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:09:59.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:09:59.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:00.499 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:00.496+0000 7f41cba347c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:10:00.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:00.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:00.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:10:00.543 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:10:00.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:00.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:01.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:02.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:02.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:02.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:10:02.272 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:10:02.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:02.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:02.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:03.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:03.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:03.782 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:10:03.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:10:03.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:03.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:04.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:05.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:05.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:05.390 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:10:05.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:10:05.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:05.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:05.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:06.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:06.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:06.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:10:06.846 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:10:06.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:06.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:07.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:08.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:08.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:08.353 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:10:08.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:10:08.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:08.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:08.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:09.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:09.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:09.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:10:09.830 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:10:09.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:09.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:10.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:11.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:11.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:11.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:10:11.338 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:10:11.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:11.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:11.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:12.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:12.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:12.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:10:12.798 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:10:12.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:12.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:13.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:14.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:14.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:14.892 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:10:14.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:10:14.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:14.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:15.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:16.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:16.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:16.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:10:16.391 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:10:16.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:16.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:17.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:18.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:18.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:10:18.169 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:10:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:18.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:19.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:19.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:19.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:10:19.950 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:10:19.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:19.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:21.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:21.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:21.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:10:21.996 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:10:21.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:21.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:22.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:23.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:23.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:23.906 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:10:23.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:10:23.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:23.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:24.413 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4130405402,v1:127.0.0.1:6819/4130405402] [v2:127.0.0.1:6820/4130405402,v1:127.0.0.1:6821/4130405402] exists,up a36fbba2-bb79-4917-9677-18ee52bc8b44 2024-09-16T15:10:24.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:10:24.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:10:24.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:10:24.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-09-16T15:10:25.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-09-16T15:10:25.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2024-09-16T15:10:25.619 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T15:10:25.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:10:26.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-09-16T15:10:27.887 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 1 2024-09-16T15:10:27.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-09-16T15:10:29.225 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 2 2024-09-16T15:10:29.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-09-16T15:10:34.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-09-16T15:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-09-16T15:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-16T15:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:10:34.249 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:10:34.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:34.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:10:34.707 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3246432608,v1:127.0.0.1:6801/3246432608] [v2:127.0.0.1:6802/3246432608,v1:127.0.0.1:6803/3246432608] exists,up bf730696-dd39-4ea4-b84b-4c7aa370fdf2 2024-09-16T15:10:34.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:10:34.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:10:34.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:10:34.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-09-16T15:10:34.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-09-16T15:10:34.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:10:34.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:10:34.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:10:34.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:10:34.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-16T15:10:34.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:10:34.709 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:10:34.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:34.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:10:35.180 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2333741688,v1:127.0.0.1:6811/2333741688] [v2:127.0.0.1:6812/2333741688,v1:127.0.0.1:6813/2333741688] exists,up 017bbcbb-6378-4adf-999e-386589c35e62 2024-09-16T15:10:35.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:10:35.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:10:35.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:10:35.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-09-16T15:10:35.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-09-16T15:10:35.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:10:35.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:10:35.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:10:35.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:10:35.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-16T15:10:35.182 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:10:35.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:10:35.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:10:35.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:35.617 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4130405402,v1:127.0.0.1:6819/4130405402] [v2:127.0.0.1:6820/4130405402,v1:127.0.0.1:6821/4130405402] exists,up a36fbba2-bb79-4917-9677-18ee52bc8b44 2024-09-16T15:10:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:10:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:10:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:10:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-09-16T15:10:35.621 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:25.444+0000 7f41bc9d9640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-09-16T15:10:35.621 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:26.060+0000 7f41bc9d9640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-09-16T15:10:35.621 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:26.516+0000 7f41bc9d9640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-09-16T15:10:35.621 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:27.524+0000 7f41ba9d5640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-09-16T15:10:35.621 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:27.880+0000 7f41bc9d9640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-16T15:10:35.621 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:28.395+0000 7f41bc9d9640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-16T15:10:35.622 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:28.871+0000 7f41ba9d5640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-09-16T15:10:35.622 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:28.875+0000 7f41ba9d5640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-09-16T15:10:35.622 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:29.215+0000 7f41bc9d9640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-09-16T15:10:35.622 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:29.655+0000 7f41bc9d9640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-16T15:10:35.622 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:29.659+0000 7f41bc9d9640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-16T15:10:35.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-09-16T15:10:35.624 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:25.444+0000 7f41bc9d9640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-09-16T15:10:35.624 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:26.064+0000 7f41bc9d9640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-09-16T15:10:35.624 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:26.516+0000 7f41bc9d9640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-09-16T15:10:35.624 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:27.524+0000 7f41ba9d5640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-09-16T15:10:35.624 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:27.880+0000 7f41bc9d9640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-16T15:10:35.624 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:28.395+0000 7f41bc9d9640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-09-16T15:10:35.625 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:28.875+0000 7f41ba9d5640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-09-16T15:10:35.625 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:28.875+0000 7f41ba9d5640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-09-16T15:10:35.625 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:29.219+0000 7f41bc9d9640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-09-16T15:10:35.625 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:29.659+0000 7f41bc9d9640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-16T15:10:35.625 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:10:29.663+0000 7f41bc9d9640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-16T15:10:35.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-09-16T15:10:35.625 INFO:tasks.workunit.client.0.smithi039.stdout:success 2024-09-16T15:10:35.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-09-16T15:10:35.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2024-09-16T15:10:35.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-16T15:10:36.257 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' does not exist 2024-09-16T15:10:36.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-09-16T15:10:36.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:10:36.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:10:36.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:10:36.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:10:36.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:10:41.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:10:41.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-09-16T15:10:41.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-16T15:10:41.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:10:41.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-16T15:10:41.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:10:41.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:10:41.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:10:41.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:10:41.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:10:41.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:10:41.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:10:41.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:10:41.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:10:41.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:10:41.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:10:41.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:10:41.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:10:41.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:10:41.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:10:41.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:10:41.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:10:41.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:10:41.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-16T15:10:41.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:10:41.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:10:41.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21279 2024-09-16T15:10:41.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21279 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/bad-inc-map 0 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:10:41.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:10:41.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:10:41.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:10:41.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:10:41.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:10:41.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:10:41.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:10:41.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:10:41.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:10:41.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:10:41.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:10:41.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:10:41.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:10:41.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T15:10:41.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-16T15:10:41.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:10:41.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:10:41.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21279 2024-09-16T15:10:41.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21279 2024-09-16T15:10:41.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:10:41.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:10:41.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T15:10:41.634 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T15:10:41.634 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T15:10:41.687 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-09-16T15:10:41.687 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-09-16T15:10:41.742 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/divergent-priors 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-09-16T15:10:41.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-09-16T15:10:41.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-09-16T15:10:41.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-09-16T15:10:41.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-09-16T15:10:41.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-09-16T15:10:41.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-09-16T15:10:41.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-09-16T15:10:41.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-09-16T15:10:41.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-09-16T15:10:41.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-09-16T15:10:41.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-09-16T15:10:41.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-09-16T15:10:41.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-09-16T15:10:41.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-09-16T15:10:41.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-09-16T15:10:41.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-09-16T15:10:41.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-09-16T15:10:41.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-09-16T15:10:41.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-09-16T15:10:41.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-09-16T15:10:41.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-09-16T15:10:41.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-09-16T15:10:41.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-09-16T15:10:41.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-09-16T15:10:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none ' 2024-09-16T15:10:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-16T15:10:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T15:10:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-09-16T15:10:41.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:10:41.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-09-16T15:10:41.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T15:10:41.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_divergent_2 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_divergent_3 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_divergent_ec' 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:10:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:10:41.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:10:41.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:10:41.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:10:41.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:10:41.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:10:41.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:10:41.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:10:41.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:10:41.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:10:41.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:10:41.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:10:41.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:10:41.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:10:41.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:10:41.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-16T15:10:41.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:10:41.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:10:41.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:10:41.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27113 2024-09-16T15:10:41.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:10:41.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:10:41.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-16T15:10:41.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:10:41.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:10:41.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:10:41.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27113 2024-09-16T15:10:41.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:10:41.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T15:10:41.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T15:10:41.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:10:41.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-16T15:10:41.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-09-16T15:10:41.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-09-16T15:10:41.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-09-16T15:10:41.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-09-16T15:10:41.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-09-16T15:10:41.776 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-09-16T15:10:41.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-09-16T15:10:41.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-09-16T15:10:41.787 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:10:41.787 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:10:41.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-09-16T15:10:41.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-16T15:10:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:10:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:10:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:10:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-16T15:10:41.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-16T15:10:42.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:10:42.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:10:42.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:10:42.188 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:10:42.188 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:10:42.188 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:10:42.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:10:42.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:10:42.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:10:42.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:10:42.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:10:42.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:10:42.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:10:42.392 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:10:42.393 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:10:42.393 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:10:42.393 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:10:42.393 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:10:42.393 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:10:42.393 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:10:42.393 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-mon.a.asok 2024-09-16T15:10:42.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:10:42.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27113/ceph-mon.a.asok config get fsid 2024-09-16T15:10:42.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:10:42.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:10:42.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:10:42.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:10:42.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:10:42.512 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:10:42.512 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:10:42.512 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:10:42.513 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:10:42.513 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:10:42.513 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:10:42.513 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-mon.a.asok 2024-09-16T15:10:42.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:10:42.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27113/ceph-mon.a.asok config get mon_host 2024-09-16T15:10:42.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-09-16T15:10:42.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-16T15:10:42.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:10:42.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:10:42.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:10:42.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-16T15:10:42.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:10:43.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:10:43.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:10:43.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:10:43.551 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:10:43.551 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:10:43.551 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:10:43.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:10:43.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:10:43.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:10:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-16T15:10:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-09-16T15:10:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:10:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:10:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:10:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:10:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-16T15:10:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:10:43.616 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:10:43.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:10:43.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:10:43.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:10:43.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:10:43.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:10:43.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:10:43.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:10:43.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:10:43.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:10:43.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:10:43.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:10:43.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:10:43.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:10:43.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-16T15:10:43.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:10:43.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=97db7296-11b6-44b2-977a-f47d0d7f80a3 2024-09-16T15:10:43.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 97db7296-11b6-44b2-977a-f47d0d7f80a3' 2024-09-16T15:10:43.619 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 97db7296-11b6-44b2-977a-f47d0d7f80a3 2024-09-16T15:10:43.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:10:43.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzSuhmE1OQJhAAfSXR7BvxrjzQqIEOV4Pi/Q== 2024-09-16T15:10:43.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzSuhmE1OQJhAAfSXR7BvxrjzQqIEOV4Pi/Q=="}' 2024-09-16T15:10:43.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 97db7296-11b6-44b2-977a-f47d0d7f80a3 -i td/divergent-priors/0/new.json 2024-09-16T15:10:43.994 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:10:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-16T15:10:44.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzSuhmE1OQJhAAfSXR7BvxrjzQqIEOV4Pi/Q== --osd-uuid 97db7296-11b6-44b2-977a-f47d0d7f80a3 2024-09-16T15:10:44.054 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:44.047+0000 7fba38c477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:10:44.056 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:44.051+0000 7fba38c477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:10:44.058 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:44.051+0000 7fba38c477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:10:44.058 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:44.051+0000 7fba38c477c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:10:44.059 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:44.051+0000 7fba38c477c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:10:44.059 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:44.051+0000 7fba38c477c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:10:44.095 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:44.091+0000 7fba38c477c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-16T15:10:50.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-16T15:10:50.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:10:50.231 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:10:50.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:10:50.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:10:50.851 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:10:50.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:10:50.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:10:50.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:10:50.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:10:50.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:10:50.893 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:50.891+0000 7fa4015237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:10:50.906 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:50.903+0000 7fa4015237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:10:50.908 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:50.907+0000 7fa4015237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:10:51.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:10:51.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:10:51.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:10:51.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:10:51.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:10:51.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:51.283 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:10:51.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:10:51.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:51.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:10:51.489 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:51.487+0000 7fa4015237c0 -1 Falling back to public interface 2024-09-16T15:10:51.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:52.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:52.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:52.716 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:10:52.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:10:52.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:52.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:10:53.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:54.046 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:10:54.042+0000 7fa4015237c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:10:54.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:54.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:54.144 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:10:54.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:10:54.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:54.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:10:54.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:55.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:55.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:55.637 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:10:55.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:10:55.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:55.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:10:56.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:57.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:57.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:57.142 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:10:57.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:10:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:10:57.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:10:58.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:10:58.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:10:58.598 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:10:58.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:10:58.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:10:58.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:10:59.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:00.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:00.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:00.106 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:11:00.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:11:00.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:00.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:00.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:01.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:01.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:01.541 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:11:01.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:11:01.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:01.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:02.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:03.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:03.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:03.082 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:11:03.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:11:03.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:03.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:03.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:04.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:04.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:04.555 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:11:04.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:11:04.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:04.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:05.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:06.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:06.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:06.088 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:11:06.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:11:06.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:06.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:06.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:07.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:07.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:07.686 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:11:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:11:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:08.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:09.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:09.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:09.262 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:11:09.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:11:09.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:09.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:10.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:11.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:11.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:11.038 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:11:11.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:11:11.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:11.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:12.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:13.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:13.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:13.016 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:11:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:11:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:13.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:14.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:14.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:14.655 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:11:14.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:11:14.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:14.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:15.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:16.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:16.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:16.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:11:16.531 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:11:16.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:16.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:17.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:18.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:18.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:18.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:11:18.100 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:11:18.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:18.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:11:18.566 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2758430011,v1:127.0.0.1:6803/2758430011] [v2:127.0.0.1:6804/2758430011,v1:127.0.0.1:6805/2758430011] exists,up 97db7296-11b6-44b2-977a-f47d0d7f80a3 2024-09-16T15:11:18.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:11:18.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:11:18.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:11:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:11:18.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:11:18.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:11:18.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:11:18.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:11:18.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:11:18.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:11:18.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:11:18.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e3ebf1d1-2f53-4bdd-949a-245697f0ce67 2024-09-16T15:11:18.573 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 e3ebf1d1-2f53-4bdd-949a-245697f0ce67 2024-09-16T15:11:18.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e3ebf1d1-2f53-4bdd-949a-245697f0ce67' 2024-09-16T15:11:18.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:11:18.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCWSuhmeJRbIxAAoBeCcQmBFqko51wpzuAQ1g== 2024-09-16T15:11:18.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCWSuhmeJRbIxAAoBeCcQmBFqko51wpzuAQ1g=="}' 2024-09-16T15:11:18.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e3ebf1d1-2f53-4bdd-949a-245697f0ce67 -i td/divergent-priors/1/new.json 2024-09-16T15:11:19.170 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:11:19.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-16T15:11:19.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWSuhmeJRbIxAAoBeCcQmBFqko51wpzuAQ1g== --osd-uuid e3ebf1d1-2f53-4bdd-949a-245697f0ce67 2024-09-16T15:11:19.235 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:19.230+0000 7f242b52b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:11:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:19.246+0000 7f242b52b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:11:19.252 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:19.250+0000 7f242b52b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:11:19.252 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:19.250+0000 7f242b52b7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:11:19.252 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:19.250+0000 7f242b52b7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:11:19.252 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:19.250+0000 7f242b52b7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:11:19.342 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:19.338+0000 7f242b52b7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-16T15:11:25.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-16T15:11:25.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:11:25.335 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:11:25.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:11:25.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:11:25.909 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:11:25.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:11:25.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:11:25.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:11:25.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:11:25.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:11:25.951 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:25.950+0000 7f8b925377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:11:25.953 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:25.950+0000 7f8b925377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:11:25.954 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:25.950+0000 7f8b925377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:11:26.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:11:26.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:11:26.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:11:26.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:11:26.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:11:26.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:26.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:11:26.334 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:11:26.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:26.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:26.521 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:26.518+0000 7f8b925377c0 -1 Falling back to public interface 2024-09-16T15:11:26.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:27.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:27.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:27.790 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:11:27.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:11:27.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:27.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:28.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:29.234 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:11:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:11:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:29.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:29.289 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:29.286+0000 7f8b925377c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:11:29.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:30.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:30.770 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:11:30.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:30.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:11:30.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:30.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:31.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:32.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:32.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:32.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:11:32.289 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:11:32.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:32.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:32.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:33.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:33.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:33.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:11:33.946 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:11:33.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:33.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:34.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:35.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:35.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:35.388 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:11:35.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:11:35.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:35.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:35.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:36.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:36.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:36.832 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:11:36.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:11:36.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:36.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:37.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:11:38.418 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:11:38.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:38.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:38.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:39.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:39.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:39.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:11:39.990 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:11:39.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:39.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:40.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:41.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:41.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:41.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:11:41.429 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:11:41.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:41.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:41.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:42.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:42.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:42.889 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:11:42.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:11:42.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:42.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:43.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:44.703 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:11:44.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:11:44.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:44.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:45.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:46.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:46.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:46.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:11:46.685 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:11:46.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:46.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:48.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:49.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:49.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:49.029 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:11:49.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:11:49.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:49.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:50.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:51.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:51.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:51.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:11:51.029 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:11:51.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:51.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:52.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:11:53.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:11:53.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:11:53.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:11:53.040 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:11:53.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:11:53.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:11:53.583 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/494124262,v1:127.0.0.1:6811/494124262] [v2:127.0.0.1:6812/494124262,v1:127.0.0.1:6813/494124262] exists,up e3ebf1d1-2f53-4bdd-949a-245697f0ce67 2024-09-16T15:11:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:11:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:11:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:11:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-16T15:11:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-09-16T15:11:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:11:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:11:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T15:11:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:11:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:11:53.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:11:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:11:53.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:11:53.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:11:53.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:11:53.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:11:53.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-16T15:11:53.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:11:53.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cf3c03ff-bb7b-4cc5-86a2-4f801ba12c77 2024-09-16T15:11:53.590 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 cf3c03ff-bb7b-4cc5-86a2-4f801ba12c77 2024-09-16T15:11:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cf3c03ff-bb7b-4cc5-86a2-4f801ba12c77' 2024-09-16T15:11:53.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:11:53.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5Suhmi+fYJBAAomzH6qJFwNu9kUcvIrio5w== 2024-09-16T15:11:53.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5Suhmi+fYJBAAomzH6qJFwNu9kUcvIrio5w=="}' 2024-09-16T15:11:53.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cf3c03ff-bb7b-4cc5-86a2-4f801ba12c77 -i td/divergent-priors/2/new.json 2024-09-16T15:11:54.224 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:11:54.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-16T15:11:54.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5Suhmi+fYJBAAomzH6qJFwNu9kUcvIrio5w== --osd-uuid cf3c03ff-bb7b-4cc5-86a2-4f801ba12c77 2024-09-16T15:11:54.282 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:54.277+0000 7fb01d8317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:11:54.304 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:54.301+0000 7fb01d8317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:11:54.305 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:54.301+0000 7fb01d8317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:11:54.306 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:54.301+0000 7fb01d8317c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:11:54.306 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:54.301+0000 7fb01d8317c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:11:54.306 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:54.301+0000 7fb01d8317c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:11:54.404 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:11:54.401+0000 7fb01d8317c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-16T15:12:00.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-16T15:12:00.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:12:00.535 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T15:12:00.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T15:12:00.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:12:01.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T15:12:01.154 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:12:01.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:12:01.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:12:01.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:12:01.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:12:01.195 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:12:01.193+0000 7fac769ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:12:01.219 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:12:01.217+0000 7fac769ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:12:01.221 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:12:01.217+0000 7fac769ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:12:01.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T15:12:01.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:12:01.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:12:01.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:12:01.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:12:01.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:01.584 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:12:01.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:12:01.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:01.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:01.789 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:12:01.785+0000 7fac769ca7c0 -1 Falling back to public interface 2024-09-16T15:12:02.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:03.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:03.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:03.051 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:12:03.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:12:03.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:03.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:03.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:04.387 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:12:04.385+0000 7fac769ca7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:12:04.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:04.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:04.486 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:12:04.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:12:04.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:04.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:05.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:06.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:06.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:06.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:12:06.016 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:12:06.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:06.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:06.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:07.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:07.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:07.484 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:12:07.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:12:07.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:07.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:07.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:08.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:08.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:08.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:12:08.970 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:12:08.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:08.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:09.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:10.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:10.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:10.459 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:12:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:12:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:10.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:11.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:11.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:11.986 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:12:11.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:12:11.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:11.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:12.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:13.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:13.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:13.505 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:12:13.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:12:13.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:13.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:13.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:14.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:14.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:14.976 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:12:14.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:12:14.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:14.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:15.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:16.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:16.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:16.476 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:12:16.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:12:16.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:16.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:17.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:18.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:18.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:18.007 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:12:18.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:12:18.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:18.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:18.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:19.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:19.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:19.950 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:12:19.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:12:19.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:19.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:20.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:21.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:21.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:21.722 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:12:21.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:12:21.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:21.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:22.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:23.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:23.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:23.561 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:12:23.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:12:23.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:23.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:24.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:25.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:25.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:25.232 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:12:25.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:12:25.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:25.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:26.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:27.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:27.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:27.382 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:12:27.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:12:27.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:27.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:28.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:12:29.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:12:29.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:12:29.019 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:12:29.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:12:29.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:12:29.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:12:29.453 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2470969433,v1:127.0.0.1:6819/2470969433] [v2:127.0.0.1:6820/2470969433,v1:127.0.0.1:6821/2470969433] exists,up cf3c03ff-bb7b-4cc5-86a2-4f801ba12c77 2024-09-16T15:12:29.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:12:29.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:12:29.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:12:29.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-09-16T15:12:30.364 INFO:tasks.workunit.client.0.smithi039.stderr:noout is set 2024-09-16T15:12:30.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-09-16T15:12:31.235 INFO:tasks.workunit.client.0.smithi039.stderr:noin is set 2024-09-16T15:12:31.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-09-16T15:12:31.963 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T15:12:31.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-09-16T15:12:31.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-16T15:12:32.703 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T15:12:32.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:12:33.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-09-16T15:12:34.718 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 3 2024-09-16T15:12:34.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-09-16T15:12:35.470 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 2 2024-09-16T15:12:35.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-09-16T15:12:35.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:12:35.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:12:35.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:12:35.928 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:12:35.928 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:12:35.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:12:35.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:12:35.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:12:36.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836503 2024-09-16T15:12:36.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836503 2024-09-16T15:12:36.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503' 2024-09-16T15:12:36.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:12:36.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:12:36.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672976 2024-09-16T15:12:36.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672976 2024-09-16T15:12:36.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672976' 2024-09-16T15:12:36.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:12:36.383 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:12:36.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509450 2024-09-16T15:12:36.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509450 2024-09-16T15:12:36.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672976 2-64424509450' 2024-09-16T15:12:36.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:12:36.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836503 2024-09-16T15:12:36.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:12:36.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:12:36.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836503 2024-09-16T15:12:36.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:12:36.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836503 2024-09-16T15:12:36.549 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836503 2024-09-16T15:12:36.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-16T15:12:36.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:12:36.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-16T15:12:36.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:12:36.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672976 2024-09-16T15:12:36.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:12:36.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:12:36.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672976 2024-09-16T15:12:36.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:12:36.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672976 2024-09-16T15:12:36.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-16T15:12:36.978 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672976 2024-09-16T15:12:36.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:12:37.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-16T15:12:37.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:12:37.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509450 2024-09-16T15:12:37.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:12:37.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:12:37.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509450 2024-09-16T15:12:37.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:12:37.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509450 2024-09-16T15:12:37.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-16T15:12:37.416 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509450 2024-09-16T15:12:37.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:12:37.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509448 -lt 64424509450 2024-09-16T15:12:37.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:12:38.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:12:38.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:12:39.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-16T15:12:39.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-09-16T15:12:39.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:12:39.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:12:39.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:12:39.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:12:39.461 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:12:39.461 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:12:39.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:12:39.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:12:39.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:12:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:12:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:12:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:12:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:12:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:12:39.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:12:40.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:12:40.010 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:12:40.010 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:12:40.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:12:40.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:12:40.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:12:40.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836506 2024-09-16T15:12:40.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836506 2024-09-16T15:12:40.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836506' 2024-09-16T15:12:40.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:12:40.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:12:40.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672979 2024-09-16T15:12:40.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672979 2024-09-16T15:12:40.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836506 1-42949672979' 2024-09-16T15:12:40.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:12:40.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:12:40.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509453 2024-09-16T15:12:40.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509453 2024-09-16T15:12:40.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836506 1-42949672979 2-64424509453' 2024-09-16T15:12:40.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:12:40.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836506 2024-09-16T15:12:40.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:12:40.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:12:40.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836506 2024-09-16T15:12:40.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:12:40.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836506 2024-09-16T15:12:40.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-16T15:12:40.509 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836506 2024-09-16T15:12:40.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:12:40.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-16T15:12:40.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:12:40.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672979 2024-09-16T15:12:40.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:12:40.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:12:40.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672979 2024-09-16T15:12:40.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:12:40.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672979 2024-09-16T15:12:40.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-16T15:12:40.917 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672979 2024-09-16T15:12:40.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:12:41.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-16T15:12:41.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:12:41.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:12:41.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509453 2024-09-16T15:12:41.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:12:41.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509453 2024-09-16T15:12:41.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:12:41.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509453 2024-09-16T15:12:41.410 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509453 2024-09-16T15:12:41.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-16T15:12:41.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:12:41.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509451 -lt 64424509453 2024-09-16T15:12:41.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:12:42.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:12:42.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:12:43.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-16T15:12:43.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:12:43.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:12:43.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:12:43.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:12:43.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:12:43.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:12:43.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:12:43.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:12:43.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:12:43.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:12:43.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:12:44.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T15:12:44.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:12:44.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:12:44.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:12:44.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T15:12:44.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:12:44.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:12:44.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-09-16T15:12:44.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-09-16T15:12:45.356 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:12:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-09-16T15:12:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-09-16T15:12:45.373 INFO:tasks.workunit.client.0.smithi039.stdout:primary and soon to be divergent is 1 2024-09-16T15:12:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-09-16T15:12:45.767 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:12:45.767 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T15:12:34.069704+0000 0'0 26:12 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:02:52.206949+0000 0 0 2024-09-16T15:12:45.767 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:12:45.767 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:12:45.767 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:12:45.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-09-16T15:12:45.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-16T15:12:45.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-09-16T15:12:45.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-09-16T15:12:45.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-16T15:12:45.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-09-16T15:12:45.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-09-16T15:12:45.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-16T15:12:45.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-09-16T15:12:45.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-09-16T15:12:45.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-09-16T15:12:45.784 INFO:tasks.workunit.client.0.smithi039.stdout:writing initial objects 2024-09-16T15:12:45.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-09-16T15:12:45.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:45.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-09-16T15:12:45.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:45.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-09-16T15:12:46.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:46.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-09-16T15:12:46.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:46.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-09-16T15:12:46.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:46.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-09-16T15:12:46.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:46.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-09-16T15:12:46.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:46.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-09-16T15:12:46.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:46.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-09-16T15:12:47.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:47.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-09-16T15:12:47.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:47.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-09-16T15:12:47.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:47.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-09-16T15:12:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:47.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-09-16T15:12:47.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:47.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-09-16T15:12:47.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:47.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-09-16T15:12:48.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:48.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-09-16T15:12:48.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:48.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-09-16T15:12:48.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:48.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-09-16T15:12:48.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:48.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-09-16T15:12:48.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:48.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-09-16T15:12:49.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:49.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-09-16T15:12:49.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:49.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-09-16T15:12:49.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:49.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-09-16T15:12:49.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:49.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-09-16T15:12:49.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:49.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-09-16T15:12:49.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:49.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-09-16T15:12:50.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:50.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-09-16T15:12:50.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:50.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-09-16T15:12:50.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:50.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-09-16T15:12:50.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:50.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-09-16T15:12:50.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:50.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-09-16T15:12:50.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:50.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-09-16T15:12:51.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:51.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-09-16T15:12:51.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:51.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-09-16T15:12:51.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:51.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-09-16T15:12:51.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:51.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-09-16T15:12:51.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:51.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-09-16T15:12:51.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:51.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-09-16T15:12:52.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:52.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-09-16T15:12:52.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:52.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-09-16T15:12:52.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:52.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-09-16T15:12:52.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:52.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-09-16T15:12:52.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:52.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-09-16T15:12:52.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:52.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-09-16T15:12:53.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:53.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-09-16T15:12:53.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:53.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-09-16T15:12:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-09-16T15:12:53.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:53.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-09-16T15:12:53.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:53.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-09-16T15:12:53.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:53.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-09-16T15:12:54.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:54.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-09-16T15:12:54.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:54.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-09-16T15:12:54.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:54.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-09-16T15:12:54.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:54.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-09-16T15:12:54.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:54.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-09-16T15:12:54.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:54.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-09-16T15:12:55.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:55.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-09-16T15:12:55.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:55.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-09-16T15:12:55.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:55.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-09-16T15:12:55.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:55.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-09-16T15:12:55.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:55.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-09-16T15:12:55.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:55.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-09-16T15:12:56.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:56.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-09-16T15:12:56.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:56.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-09-16T15:12:56.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:56.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-09-16T15:12:56.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:56.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-09-16T15:12:56.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:56.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-09-16T15:12:57.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:57.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-09-16T15:12:57.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:57.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-09-16T15:12:57.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:57.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-09-16T15:12:57.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:57.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-09-16T15:12:57.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:57.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-09-16T15:12:57.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:57.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-09-16T15:12:57.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:57.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-09-16T15:12:58.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:58.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-09-16T15:12:58.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:58.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-09-16T15:12:58.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:58.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-09-16T15:12:58.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:58.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-09-16T15:12:58.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:58.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-09-16T15:12:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-09-16T15:12:59.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:59.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-09-16T15:12:59.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:59.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-09-16T15:12:59.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:59.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-09-16T15:12:59.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:59.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-09-16T15:12:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:12:59.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-09-16T15:13:00.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:00.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-09-16T15:13:00.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:00.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-09-16T15:13:00.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:00.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-09-16T15:13:00.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:00.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-09-16T15:13:00.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:00.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-09-16T15:13:01.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:01.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-09-16T15:13:01.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:01.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-09-16T15:13:01.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:01.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-09-16T15:13:01.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:01.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-09-16T15:13:01.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:01.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-09-16T15:13:01.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:01.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-09-16T15:13:02.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:02.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-09-16T15:13:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-09-16T15:13:02.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:02.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-09-16T15:13:02.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:02.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-09-16T15:13:02.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-16T15:13:02.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-16T15:13:02.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T15:13:02.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-09-16T15:13:02.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:13:02.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:13:02.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:13:02.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T15:13:02.896 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:13:02.896 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:13:02.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:13:02.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:13:02.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:13:02.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T15:13:02.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:13:02.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:13:02.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:13:02.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:13:02.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:13:03.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:13:03.420 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:13:03.420 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:13:03.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:13:03.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:13:03.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:13:03.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836513 2024-09-16T15:13:03.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836513 2024-09-16T15:13:03.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836513' 2024-09-16T15:13:03.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:13:03.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:13:03.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672985 2024-09-16T15:13:03.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672985 2024-09-16T15:13:03.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836513 1-42949672985' 2024-09-16T15:13:03.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:13:03.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:13:03.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509459 2024-09-16T15:13:03.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509459 2024-09-16T15:13:03.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836513 1-42949672985 2-64424509459' 2024-09-16T15:13:03.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:13:03.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836513 2024-09-16T15:13:03.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:13:03.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:13:03.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:13:03.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836513 2024-09-16T15:13:03.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836513 2024-09-16T15:13:03.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-16T15:13:03.947 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836513 2024-09-16T15:13:03.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:13:04.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836510 -lt 21474836513 2024-09-16T15:13:04.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:13:05.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:13:05.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:13:05.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-16T15:13:05.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:13:05.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672985 2024-09-16T15:13:05.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:13:05.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:13:05.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672985 2024-09-16T15:13:05.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:13:05.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672985 2024-09-16T15:13:05.801 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672985 2024-09-16T15:13:05.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-16T15:13:05.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:13:06.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672986 -lt 42949672985 2024-09-16T15:13:06.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:13:06.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509459 2024-09-16T15:13:06.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:13:06.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:13:06.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509459 2024-09-16T15:13:06.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:13:06.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509459 2024-09-16T15:13:06.339 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509459 2024-09-16T15:13:06.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-16T15:13:06.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:13:06.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509460 -lt 64424509459 2024-09-16T15:13:06.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:13:06.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:13:06.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:13:07.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:13:07.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:13:07.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:13:07.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:13:07.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:13:07.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:13:07.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:13:07.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:13:07.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T15:13:07.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:13:07.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:13:07.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:13:08.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T15:13:08.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:13:08.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:13:08.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-09-16T15:13:08.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-16T15:13:08.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-16T15:13:08.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T15:13:08.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-16T15:13:08.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-09-16T15:13:08.915 INFO:tasks.workunit.client.0.smithi039.stdout:blackholing osds 0 2 2024-09-16T15:13:08.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-09-16T15:13:08.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-09-16T15:13:09.365 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:13:09.366 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-16T15:12:34.069704+0000 28'100 28:112 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:02:52.206949+0000 0 0 2024-09-16T15:13:09.366 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:13:09.366 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:13:09.366 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:13:09.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-16T15:13:09.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-09-16T15:13:09.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T15:13:09.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T15:13:09.385 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:13:09.386 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:13:09.386 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:13:09.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.0.asok 2024-09-16T15:13:09.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-16T15:13:09.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-16T15:13:09.498 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:13:09.498 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-16T15:13:09.498 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:13:09.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-16T15:13:09.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-09-16T15:13:09.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T15:13:09.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T15:13:09.513 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:13:09.513 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:13:09.513 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:13:09.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.2.asok 2024-09-16T15:13:09.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-16T15:13:09.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-16T15:13:09.627 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:13:09.627 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-16T15:13:09.627 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:13:09.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-09-16T15:13:09.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-09-16T15:13:09.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-09-16T15:13:09.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-09-16T15:13:09.643 INFO:tasks.workunit.client.0.smithi039.stdout:writing divergent object 2024-09-16T15:13:09.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-16T15:13:09.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-09-16T15:13:09.644 INFO:tasks.workunit.client.0.smithi039.stdout:create missing divergent object 2024-09-16T15:13:09.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-09-16T15:13:09.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T15:13:09.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=test 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=existing_99 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/divergent-priors 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds test existing_99 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-09-16T15:13:09.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2024-09-16T15:13:09.647 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2024-09-16T15:13:09.647 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T15:13:10.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-16T15:13:10.091 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-09-16T15:13:10.091 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:13:10.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2024-09-16T15:13:10.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('1' '0' '2') 2024-09-16T15:13:10.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T15:13:10.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T15:13:10.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T15:13:10.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T15:13:10.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/divergent-priors/1/type 2024-09-16T15:13:10.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T15:13:10.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T15:13:10.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T15:13:10.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T15:13:10.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T15:13:10.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T15:13:10.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T15:13:10.094 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T15:13:10.094 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T15:13:10.095 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T15:13:10.095 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:13:10.095 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:13:10.095 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:13:10.096 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.1.asok 2024-09-16T15:13:10.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.27113/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T15:13:10.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T15:13:10.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T15:13:10.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T15:13:10.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T15:13:10.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T15:13:10.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T15:13:10.209 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:13:10.209 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:13:10.209 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:13:10.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.1.asok 2024-09-16T15:13:10.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T15:13:10.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.1.asok injectdataerr test existing_99 2024-09-16T15:13:10.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-09-16T15:13:10.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-09-16T15:13:20.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-09-16T15:13:20.349 INFO:tasks.workunit.client.0.smithi039.stderr:rados: no process found 2024-09-16T15:13:20.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-09-16T15:13:20.349 INFO:tasks.workunit.client.0.smithi039.stdout:killing all the osds 2024-09-16T15:13:20.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-09-16T15:13:20.762 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:13:20.762 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 active+clean 2024-09-16T15:13:10.555330+0000 28'101 28:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:02:52.206949+0000 0 0 2024-09-16T15:13:20.762 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:13:20.762 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:13:20.763 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:13:20.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-09-16T15:13:20.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:13:20.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:13:20.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:13:20.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:13:20.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:13:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:13:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-16T15:13:20.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-09-16T15:13:21.457 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T15:13:21.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-16T15:13:21.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-09-16T15:13:22.109 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-16T15:13:22.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-16T15:13:22.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-09-16T15:13:22.716 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-16T15:13:22.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-16T15:13:22.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-09-16T15:13:23.299 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T15:13:23.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-16T15:13:23.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-09-16T15:13:23.898 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already out. 2024-09-16T15:13:23.914 INFO:tasks.workunit.client.0.smithi039.stdout:bringing up non_divergent 0 2 2024-09-16T15:13:23.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-09-16T15:13:23.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-09-16T15:13:24.343 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:13:24.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 stale+active+clean 2024-09-16T15:13:10.555330+0000 28'101 28:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:02:52.206949+0000 0 0 2024-09-16T15:13:24.343 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:13:24.343 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:13:24.343 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:13:24.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-16T15:13:24.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-09-16T15:13:24.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:13:24.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:13:24.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:13:24.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:13:24.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:13:24.364 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:13:24.364 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:13:24.364 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:13:24.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:13:24.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:13:24.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:13:24.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:13:24.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:13:24.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:13:24.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:13:24.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:13:24.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:13:24.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:13:24.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:13:24.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-16T15:13:24.366 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:13:24.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:13:24.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:13:24.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-16T15:13:24.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:13:24.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:13:24.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:13:24.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:13:24.400 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:13:24.396+0000 7fefd51f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:13:24.403 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:13:24.400+0000 7fefd51f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:13:24.404 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:13:24.400+0000 7fefd51f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:13:24.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:13:24.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:13:24.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:13:24.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:13:24.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:13:24.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:13:24.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:13:24.811 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:13:24.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:13:24.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:13:24.968 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:13:24.964+0000 7fefd51f87c0 -1 Falling back to public interface 2024-09-16T15:13:25.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:13:26.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:13:26.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:13:26.286 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:13:26.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:13:26.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:13:26.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:13:26.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:13:27.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:13:27.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:13:27.732 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:13:27.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:13:27.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:13:27.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:13:27.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:13:27.888+0000 7fefd51f87c0 -1 osd.0 28 log_to_monitors true 2024-09-16T15:13:28.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:13:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:13:29.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:13:29.217 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:13:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:13:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:13:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:13:29.703 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/2870903764,v1:127.0.0.1:6803/2870903764] [v2:127.0.0.1:6804/2870903764,v1:127.0.0.1:6805/2870903764] exists,up 97db7296-11b6-44b2-977a-f47d0d7f80a3 2024-09-16T15:13:29.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:13:29.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:13:29.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:13:29.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:13:29.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:13:29.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:13:29.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:13:29.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:13:29.707 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:13:29.707 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:13:29.707 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:13:29.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:13:29.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:13:29.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:13:29.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:13:29.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:13:29.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:13:29.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:13:29.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:13:29.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:13:29.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:13:29.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:13:29.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-16T15:13:29.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T15:13:29.709 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:13:29.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:13:29.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-16T15:13:29.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T15:13:29.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:13:29.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:13:29.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:13:29.748 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:13:29.744+0000 7f4375ec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:13:29.765 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:13:29.760+0000 7f4375ec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:13:29.766 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:13:29.764+0000 7f4375ec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:13:30.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T15:13:30.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:13:30.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:13:30.188 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:13:30.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:13:30.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:13:30.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:13:30.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:13:30.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:13:30.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:13:30.324 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:13:30.320+0000 7f4375ec67c0 -1 Falling back to public interface 2024-09-16T15:13:30.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:13:31.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:13:31.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:13:31.632 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:13:31.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:13:31.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:13:31.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:13:32.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:13:33.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:13:33.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:13:33.066 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:13:33.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:13:33.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:13:33.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:13:33.401 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:13:33.396+0000 7f4375ec67c0 -1 osd.2 28 log_to_monitors true 2024-09-16T15:13:33.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:13:34.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:13:34.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:13:34.592 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:13:34.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:13:34.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:13:34.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:13:35.074 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/967105325,v1:127.0.0.1:6811/967105325] [v2:127.0.0.1:6812/967105325,v1:127.0.0.1:6813/967105325] exists,up cf3c03ff-bb7b-4cc5-86a2-4f801ba12c77 2024-09-16T15:13:35.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:13:35.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:13:35.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:13:35.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-16T15:13:35.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-09-16T15:13:35.727 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already in. 2024-09-16T15:13:35.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-16T15:13:35.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-09-16T15:13:36.704 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already in. 2024-09-16T15:13:36.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T15:13:36.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-09-16T15:13:36.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:13:36.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:13:36.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:13:36.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T15:13:36.728 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:13:36.728 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:13:36.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:13:36.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:13:36.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:13:36.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T15:13:36.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:13:36.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:13:36.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:13:36.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:13:36.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:13:37.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:13:37.302 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:13:37.302 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:13:37.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:13:37.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:13:37.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:13:37.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=150323855365 2024-09-16T15:13:37.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 150323855365 2024-09-16T15:13:37.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-150323855365' 2024-09-16T15:13:37.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:13:37.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:13:37.590 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T15:13:37.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T15:13:37.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T15:13:37.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T15:13:37.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:13:37.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:13:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=158913789955 2024-09-16T15:13:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 158913789955 2024-09-16T15:13:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-150323855365 2-158913789955' 2024-09-16T15:13:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:13:37.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-150323855365 2024-09-16T15:13:37.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:13:37.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:13:37.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-150323855365 2024-09-16T15:13:37.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:13:37.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=150323855365 2024-09-16T15:13:37.763 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 150323855365 2024-09-16T15:13:37.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-09-16T15:13:37.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:13:38.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 150323855363 -lt 150323855365 2024-09-16T15:13:38.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:13:39.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:13:39.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:13:39.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 150323855365 -lt 150323855365 2024-09-16T15:13:39.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:13:39.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:13:39.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-158913789955 2024-09-16T15:13:39.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:13:39.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-158913789955 2024-09-16T15:13:39.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:13:39.639 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 158913789955 2024-09-16T15:13:39.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=158913789955 2024-09-16T15:13:39.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2024-09-16T15:13:39.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:13:40.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 158913789955 -lt 158913789955 2024-09-16T15:13:40.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:13:40.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:13:40.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:13:40.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:13:40.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:13:40.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:13:40.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:13:40.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:13:40.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:13:40.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:13:40.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:13:41.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:13:41.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:13:41.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:13:41.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:13:41.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:13:41.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T15:13:41.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T15:13:41.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T15:13:41.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:13:41.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T15:13:41.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:13:41.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:13:41.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:13:41.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:13:41.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:13:41.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:13:41.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:13:41.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:13:42.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:13:42.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:13:42.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:13:42.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:13:42.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:13:42.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:13:42.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:13:42.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:13:42.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:13:42.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:13:42.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:13:42.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:13:42.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:13:43.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:13:43.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:13:43.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-16T15:13:43.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:13:43.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T15:13:43.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:13:43.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:13:43.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:13:43.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:13:43.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:13:43.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:13:43.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:13:43.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:13:43.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:13:43.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:13:43.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:13:43.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:13:44.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:13:44.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:13:44.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:13:44.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:13:44.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:13:44.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:13:44.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:13:44.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:13:44.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:13:45.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:13:45.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:13:45.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-16T15:13:45.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:13:45.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T15:13:45.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:13:45.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:13:45.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:13:45.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:13:45.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:13:45.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:13:45.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:13:45.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:13:45.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:13:45.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:13:45.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:13:45.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:13:46.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:13:46.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:13:46.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:13:46.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:13:46.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:13:46.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:13:46.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:13:46.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:13:46.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:13:47.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:13:47.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:13:47.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-16T15:13:47.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:13:47.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-16T15:13:47.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:13:47.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:13:47.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:13:47.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:13:47.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:13:47.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:13:47.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:13:47.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:13:48.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:13:48.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:13:48.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:13:48.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:13:48.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:13:48.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:13:48.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:13:48.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:13:48.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:13:48.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:13:48.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:13:48.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:13:48.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:13:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:13:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:13:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-16T15:13:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:13:49.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-16T15:13:51.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:13:51.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:13:51.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:13:51.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:13:51.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:13:51.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:13:51.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:13:51.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:13:51.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:13:51.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:13:51.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:13:51.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:13:51.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:13:51.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:13:51.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:13:51.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:13:51.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:13:51.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:13:51.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:13:51.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:13:51.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:13:52.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:13:52.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:13:52.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-16T15:13:52.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:13:52.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-16T15:13:55.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:13:55.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:13:55.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:13:55.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:13:55.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:13:55.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:13:55.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:13:55.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:13:56.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:13:56.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:13:56.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:13:56.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:13:56.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:13:56.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:13:56.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:13:56.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:13:56.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:13:56.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:13:56.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:13:56.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:13:56.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:13:57.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:13:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:13:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-16T15:13:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:13:57.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-16T15:14:03.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:14:03.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:14:03.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:14:03.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:14:03.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:14:03.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:14:03.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:14:03.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:14:04.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:14:04.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:14:04.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:14:04.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:14:04.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:14:04.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:14:04.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:14:04.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:14:04.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:14:04.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:14:04.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:14:04.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:14:04.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:14:05.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:14:05.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:14:05.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-16T15:14:05.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:14:05.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-16T15:14:12.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:14:12.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:14:12.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:14:12.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:14:12.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:14:12.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:14:12.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:14:12.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:14:13.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:14:13.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:14:13.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:14:13.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:14:13.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:14:13.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:14:13.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:14:13.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:14:13.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:14:13.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:14:13.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:14:13.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:14:13.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:14:14.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:14:14.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:14:14.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-16T15:14:14.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T15:14:14.779 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:14:14.779 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "98b0cdcb-8782-4259-9d7f-3259684d308a", 2024-09-16T15:14:14.779 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "18.2.4-845-ge4e227b3", 2024-09-16T15:14:14.779 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "e4e227b37858f5109138494e01e5ba1ba7b8ffe8", 2024-09-16T15:14:14.779 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-16T15:14:14.776739+0000", 2024-09-16T15:14:14.779 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-16T15:14:14.779 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-16T15:14:14.779 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-16T15:14:14.779 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-16T15:14:14.779 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-16T15:14:14.779 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:14:14.780 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:14:14.781 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.782 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-16T15:14:14.783 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:10:41.818345Z", 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:10:41.818345Z", 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 18, 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "reef", 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-16T15:14:14.784 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: "reef" 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-16T15:14:14.785 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.786 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 42, 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:10:42.732451+0000", 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:13:36.755138+0000", 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-16T15:13:33.779155+0000", 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-16T15:13:36.209075+0000", 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-16T15:14:14.787 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 9, 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-16T15:14:14.788 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "reef", 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-16T15:12:32.308048+0000", 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-16T15:14:14.789 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 1, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 1, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 1, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 1, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-16T15:14:14.790 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "28", 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-16T15:14:14.791 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-16T15:14:14.792 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-16T15:14:14.793 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 2.9900000095367432, 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 2.9900000095367432, 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 0.67000001668930054, 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 2, 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 2, 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.794 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "97db7296-11b6-44b2-977a-f47d0d7f80a3", 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 28, 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 35, 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 41, 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 29, 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:14:14.795 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2870903764 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2870903764 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.796 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2870903764 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2870903764 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.797 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2870903764 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2870903764 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.798 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2870903764 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2870903764 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/2870903764", 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/2870903764", 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2870903764", 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2870903764", 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:14:14.799 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "e3ebf1d1-2f53-4bdd-949a-245697f0ce67", 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 23, 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 30, 2024-09-16T15:14:14.800 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 494124262 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 494124262 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.801 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 494124262 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 494124262 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.802 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 494124262 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 494124262 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.803 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 494124262 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 494124262 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/494124262", 2024-09-16T15:14:14.804 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/494124262", 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/494124262", 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/494124262", 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "cf3c03ff-bb7b-4cc5-86a2-4f801ba12c77", 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-16T15:14:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 28, 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 37, 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 0, 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 31, 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 967105325 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 967105325 2024-09-16T15:14:14.806 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 967105325 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:14:14.807 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 967105325 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 967105325 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.808 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 967105325 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 967105325 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:14:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 967105325 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/967105325", 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/967105325", 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/967105325", 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/967105325", 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-16T15:14:14.810 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:13:21.231439+0000", 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:11:17.116015+0000", 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:13:21.853854+0000", 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:14:14.811 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:11:52.498619+0000", 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:13:22.514769+0000", 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:12:28.018769+0000", 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.812 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.813 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.814 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/2870903764,v1:127.0.0.1:6805/2870903764]", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:14:14.815 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:10:49.561223Z", 2024-09-16T15:14:14.816 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/2870903764,v1:127.0.0.1:6803/2870903764]", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2870903764,v1:127.0.0.1:6809/2870903764]", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2870903764,v1:127.0.0.1:6807/2870903764]", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:14:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/494124262,v1:127.0.0.1:6813/494124262]", 2024-09-16T15:14:14.818 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:14:14.819 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:11:24.579811Z", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:14:14.820 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/494124262,v1:127.0.0.1:6811/494124262]", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/494124262,v1:127.0.0.1:6817/494124262]", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/494124262,v1:127.0.0.1:6815/494124262]", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:14:14.821 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/967105325,v1:127.0.0.1:6813/967105325]", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:14:14.822 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:11:59.857369Z", 2024-09-16T15:14:14.823 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/967105325,v1:127.0.0.1:6811/967105325]", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/967105325,v1:127.0.0.1:6817/967105325]", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/967105325,v1:127.0.0.1:6815/967105325]", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:14:14.824 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [] 2024-09-16T15:14:14.825 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 35 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 37 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-16T15:14:14.826 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 42, 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:14:14.827 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-16T15:14:14.828 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-16T15:14:14.829 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-16T15:14:14.830 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:14:14.831 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:14:14.832 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:14:14.833 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:14:14.834 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.835 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.836 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-16T15:14:14.837 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-16T15:14:14.838 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 0, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-16T15:14:14.839 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:14:14.840 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-16T15:14:14.840 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-16T15:14:14.840 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-16T15:14:14.840 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-16T15:14:14.840 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-16T15:14:14.840 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-16T15:14:14.840 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-16T15:14:14.840 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T15:14:14.840 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-16T15:14:14.840 INFO:tasks.workunit.client.0.smithi039.stderr:report 1921312154 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2" 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-16T15:14:14.841 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 1, 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-16T15:14:14.842 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 61700, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 100, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 100, 2024-09-16T15:14:14.843 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:14:14.844 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-16T15:14:14.845 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 16, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 16, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:14:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83628, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2116, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489172, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036912128, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:14:14.847 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2166784, 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 461582, 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 1, 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:14:14.848 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:14:14.849 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83628, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2116, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489172, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036912128, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2166784, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 461582, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:14:14.850 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 1, 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T15:14:14.851 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 61700, 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-16T15:14:14.852 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 100, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 100, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:14:14.853 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:14:14.854 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1228800, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 185100, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:14:14.855 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 16, 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 16, 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 150323855373 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672989 2024-09-16T15:14:14.856 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 158913789962 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 1 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:14:14.857 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.858 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 5, 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:13:24.384210+0000", 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 95, 2024-09-16T15:14:14.859 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-16T15:14:14.860 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 382, 2024-09-16T15:14:14.860 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-16T15:14:14.860 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:14:14.860 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 527, 2024-09-16T15:14:14.860 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-16T15:14:14.860 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-16T15:14:14.860 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:14:14.860 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:14:14.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T15:14:14.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-09-16T15:14:14.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-09-16T15:14:14.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-09-16T15:14:14.861 INFO:tasks.workunit.client.0.smithi039.stdout:writing non-divergent object existing_4 2024-09-16T15:14:14.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-09-16T15:14:15.191 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:14:15.191 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-16T15:13:52.302375+0000 28'100 42:130 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:05:59.383803+0000 0 0 2024-09-16T15:14:15.192 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:14:15.192 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:14:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:14:15.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-09-16T15:14:15.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-09-16T15:14:15.302 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery 2024-09-16T15:14:15.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-09-16T15:14:15.704 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:14:15.704 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-16T15:13:52.302375+0000 28'100 42:130 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:05:59.383803+0000 0 0 2024-09-16T15:14:15.704 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:14:15.704 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:14:15.704 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:14:15.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-16T15:14:15.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-09-16T15:14:15.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T15:14:15.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T15:14:15.721 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:14:15.721 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:15.721 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:15.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.0.asok 2024-09-16T15:14:15.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-16T15:14:15.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.0.asok set_recovery_delay 100000 2024-09-16T15:14:15.832 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:14:15.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-16T15:14:15.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-09-16T15:14:15.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T15:14:15.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T15:14:15.846 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:14:15.846 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:15.846 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:15.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.2.asok 2024-09-16T15:14:15.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-16T15:14:15.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.2.asok set_recovery_delay 100000 2024-09-16T15:14:15.948 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:14:15.960 INFO:tasks.workunit.client.0.smithi039.stdout:revive divergent 1 2024-09-16T15:14:15.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-09-16T15:14:15.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-09-16T15:14:16.362 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:14:16.363 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-16T15:13:52.302375+0000 28'100 42:130 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:05:59.383803+0000 0 0 2024-09-16T15:14:16.363 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:14:16.363 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:14:16.363 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:14:16.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-09-16T15:14:17.230 INFO:tasks.workunit.client.0.smithi039.stderr:noup is set 2024-09-16T15:14:17.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-16T15:14:17.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:14:17.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:14:17.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:14:17.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:14:17.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:14:17.256 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:14:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:14:17.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:14:17.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:14:17.258 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:14:17.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:14:17.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:14:17.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-16T15:14:17.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:14:17.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:14:17.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:14:17.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:14:17.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:17.304+0000 7f89dce4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:14:17.320 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:17.316+0000 7f89dce4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:14:17.322 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:17.320+0000 7f89dce4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:14:17.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-16T15:14:17.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-09-16T15:14:17.856 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:17.852+0000 7f89dce4e7c0 -1 Falling back to public interface 2024-09-16T15:14:20.929 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:20.924+0000 7f89dce4e7c0 -1 osd.1 28 log_to_monitors true 2024-09-16T15:14:22.723 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery divergent 2024-09-16T15:14:22.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-09-16T15:14:22.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-09-16T15:14:23.144 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:14:23.145 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-16T15:13:52.302375+0000 42'101 43:133 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:05:59.383803+0000 0 0 2024-09-16T15:14:23.145 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:14:23.145 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:14:23.145 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:14:23.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-09-16T15:14:23.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T15:14:23.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T15:14:23.163 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:14:23.163 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:23.163 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:23.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.1.asok 2024-09-16T15:14:23.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-09-16T15:14:23.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.1.asok set_recovery_delay 100000 2024-09-16T15:14:23.280 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:14:23.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-09-16T15:14:24.019 INFO:tasks.workunit.client.0.smithi039.stderr:noup is unset 2024-09-16T15:14:24.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-09-16T15:14:24.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:14:24.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:14:24.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:14:24.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:14:24.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:14:24.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:14:24.038 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:14:24.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:14:24.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:14:24.551 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 35 up_thru 41 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/2870903764,v1:127.0.0.1:6803/2870903764] [v2:127.0.0.1:6804/2870903764,v1:127.0.0.1:6805/2870903764] exists,up 97db7296-11b6-44b2-977a-f47d0d7f80a3 2024-09-16T15:14:24.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:14:24.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:14:24.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:14:24.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-09-16T15:14:24.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:14:24.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:14:24.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:14:24.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:14:24.553 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:14:24.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:14:24.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:14:24.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:14:24.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:14:24.977 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/1065683714,v1:127.0.0.1:6819/1065683714] [v2:127.0.0.1:6820/1065683714,v1:127.0.0.1:6821/1065683714] exists,up e3ebf1d1-2f53-4bdd-949a-245697f0ce67 2024-09-16T15:14:24.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:14:24.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:14:24.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:14:24.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-09-16T15:14:24.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:14:24.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:14:24.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:14:24.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:14:24.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:14:24.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:14:24.979 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:14:24.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:14:24.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:14:25.436 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/967105325,v1:127.0.0.1:6811/967105325] [v2:127.0.0.1:6812/967105325,v1:127.0.0.1:6813/967105325] exists,up cf3c03ff-bb7b-4cc5-86a2-4f801ba12c77 2024-09-16T15:14:25.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:14:25.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:14:25.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:14:25.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-09-16T15:14:25.845 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:14:25.845 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-16T15:13:52.302375+0000 42'101 45:135 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:05:59.383803+0000 0 0 2024-09-16T15:14:25.846 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:14:25.846 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:14:25.846 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:14:25.861 INFO:tasks.workunit.client.0.smithi039.stdout:wait for peering 2024-09-16T15:14:25.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-09-16T15:14:25.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-09-16T15:14:26.257 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:14:26.257 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-16T15:13:52.302375+0000 42'101 45:135 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:05:59.383803+0000 0 0 2024-09-16T15:14:26.258 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:14:26.258 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:14:26.258 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:14:26.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-09-16T15:14:26.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-09-16T15:14:26.394 INFO:tasks.workunit.client.0.smithi039.stdout:killing divergent 1 2024-09-16T15:14:26.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-09-16T15:14:26.802 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:14:26.802 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 2 2 0 0 62050 0 0 17 84 17 active+recovering+degraded 2024-09-16T15:14:25.082069+0000 42'101 48:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-17T19:12:46.110277+0000 0 0 2024-09-16T15:14:26.802 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:14:26.802 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:14:26.802 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:14:26.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-09-16T15:14:26.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:14:26.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:14:26.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:14:26.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:14:26.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:14:26.928 INFO:tasks.workunit.client.0.smithi039.stdout:reviving divergent 1 2024-09-16T15:14:26.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:14:26.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-09-16T15:14:26.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-09-16T15:14:27.325 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:14:27.326 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 2 2 0 0 62050 0 0 17 84 17 active+recovering+degraded 2024-09-16T15:14:25.082069+0000 42'101 48:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-17T19:12:46.110277+0000 0 0 2024-09-16T15:14:27.326 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:14:27.326 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:14:27.326 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:14:27.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-16T15:14:27.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:14:27.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:14:27.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:14:27.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:14:27.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:14:27.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:14:27.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:14:27.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:14:27.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:14:27.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:14:27.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:14:27.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:14:27.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:14:27.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:14:27.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:14:27.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:14:27.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:14:27.344 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:14:27.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:14:27.346 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:14:27.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:14:27.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:14:27.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-16T15:14:27.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:14:27.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:14:27.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:14:27.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:14:27.385 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:27.380+0000 7f9e694207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:14:27.387 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:27.384+0000 7f9e694207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:14:27.388 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:27.384+0000 7f9e694207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:14:27.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T15:14:27.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:14:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:14:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:14:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:14:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:14:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:14:27.825 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:14:27.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:14:27.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:14:27.945 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:27.939+0000 7f9e694207c0 -1 Falling back to public interface 2024-09-16T15:14:28.261 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/1065683714,v1:127.0.0.1:6819/1065683714] [v2:127.0.0.1:6820/1065683714,v1:127.0.0.1:6821/1065683714] exists,up e3ebf1d1-2f53-4bdd-949a-245697f0ce67 2024-09-16T15:14:28.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:14:28.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:14:28.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:14:28.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-09-16T15:14:30.970 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:30.967+0000 7f9e694207c0 -1 osd.1 48 log_to_monitors true 2024-09-16T15:14:48.262 INFO:tasks.workunit.client.0.smithi039.stdout:allowing recovery 2024-09-16T15:14:48.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-09-16T15:14:48.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-09-16T15:14:48.676 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:14:48.676 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-09-16T15:14:32.879504+0000 48'102 53:150 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-17T15:16:26.127558+0000 0 0 2024-09-16T15:14:48.676 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:14:48.676 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:14:48.676 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:14:48.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-16T15:14:48.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-16T15:14:48.845 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:14:48.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-16T15:14:48.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-16T15:14:49.012 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:14:49.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-16T15:14:49.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-16T15:14:49.183 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:14:49.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-09-16T15:14:49.199 INFO:tasks.workunit.client.0.smithi039.stdout:reading divergent objects 2024-09-16T15:14:49.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-09-16T15:14:49.607 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:14:49.607 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-09-16T15:14:32.879504+0000 48'102 53:150 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:12:32.337234+0000 0'0 2024-09-16T15:12:32.337234+0000 0 0 periodic scrub scheduled @ 2024-09-17T15:16:26.127558+0000 0 0 2024-09-16T15:14:49.607 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:14:49.607 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:14:49.608 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:14:49.625 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-09-16T15:14:49.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-09-16T15:14:49.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:14:49.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-09-16T15:14:49.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:14:49.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-09-16T15:14:49.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:14:49.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-09-16T15:14:49.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:14:49.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-09-16T15:14:49.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-09-16T15:14:49.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-16T15:14:49.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-16T15:14:49.825 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:14:24.868+0000 7f89c0dd9640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4622.0:1 2024-09-16T15:13:09.677845+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2024-09-16T15:14:49.825 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:14:24.868+0000 7f89c0dd9640 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4622.0:1 2024-09-16T15:13:09.677845+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2024-09-16T15:14:49.825 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:14:24.868+0000 7f89c0dd9640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:14:49.826 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:14:24.868+0000 7f89c0dd9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'101 2024-09-16T15:14:49.826 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:14:24.868+0000 7f89c0dd9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-16T15:14:49.826 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:14:24.868+0000 7f89c0dd9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-16T15:14:49.826 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:14:24.868+0000 7f89c0dd9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'101 2024-09-16T15:14:49.826 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:14:24.868+0000 7f89c0dd9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4622.0:1 2024-09-16T15:13:09.677845+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2024-09-16T15:14:49.826 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:14:24.868+0000 7f89c0dd9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-16T15:14:49.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-16T15:14:49.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-16T15:14:49.831 INFO:tasks.workunit.client.0.smithi039.stdout:success 2024-09-16T15:14:49.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-09-16T15:14:49.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-09-16T15:14:49.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T15:14:49.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T15:14:50.356 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T15:14:50.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-09-16T15:14:50.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:14:50.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:14:50.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:14:50.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:14:50.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:14:55.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:14:55.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-16T15:14:55.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-16T15:14:55.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:14:55.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-16T15:14:55.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:14:55.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:14:55.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:14:55.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:14:55.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:14:55.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:14:55.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:14:55.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:14:55.701 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:14:55.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:14:55.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:14:55.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:14:55.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:14:55.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:14:55.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:14:55.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:14:55.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:14:55.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:14:55.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-16T15:14:55.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:14:55.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:55.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:55.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27113 2024-09-16T15:14:55.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:14:55.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:14:55.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-16T15:14:55.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-16T15:14:55.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-16T15:14:55.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-16T15:14:55.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-16T15:14:55.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:14:55.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-16T15:14:55.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:14:55.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:14:55.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:14:55.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:14:55.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:14:55.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:14:55.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:14:55.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:14:55.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:14:55.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:14:55.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:14:55.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:14:55.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:14:55.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:14:55.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:14:55.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:14:55.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:14:55.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:14:55.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-16T15:14:55.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:14:55.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:55.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:55.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27113 2024-09-16T15:14:55.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:14:55.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:14:55.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-16T15:14:55.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:14:55.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:55.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:55.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27113 2024-09-16T15:14:55.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:14:55.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T15:14:55.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:14:55.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-16T15:14:55.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-09-16T15:14:55.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-09-16T15:14:55.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-09-16T15:14:55.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-09-16T15:14:55.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-09-16T15:14:55.748 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-09-16T15:14:55.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-09-16T15:14:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-09-16T15:14:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:14:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:14:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-09-16T15:14:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-16T15:14:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:14:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:14:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:14:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-16T15:14:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-16T15:14:56.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:14:56.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:14:56.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:14:56.125 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:14:56.125 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:56.125 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:56.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:14:56.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:14:56.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:14:56.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:14:56.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:14:56.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:14:56.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:14:56.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:14:56.337 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:14:56.337 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:14:56.337 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:14:56.337 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:14:56.338 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:56.338 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:56.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-mon.a.asok 2024-09-16T15:14:56.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:14:56.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27113/ceph-mon.a.asok config get fsid 2024-09-16T15:14:56.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:14:56.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:14:56.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:14:56.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:14:56.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:14:56.455 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:14:56.455 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:14:56.455 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:14:56.455 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:14:56.455 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:56.455 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:56.456 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-mon.a.asok 2024-09-16T15:14:56.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:14:56.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27113/ceph-mon.a.asok config get mon_host 2024-09-16T15:14:56.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-09-16T15:14:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-16T15:14:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:14:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:14:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:14:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-16T15:14:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:14:57.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:14:57.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:14:57.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:14:57.488 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:14:57.488 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:57.488 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:57.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:14:57.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:14:57.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:14:57.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-16T15:14:57.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:14:57.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:14:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:14:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:14:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:14:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:14:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:14:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:14:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:14:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:14:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:14:57.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-16T15:14:57.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:14:57.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd12a0dc-6b50-4591-8f3b-ac39e6d7839e 2024-09-16T15:14:57.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fd12a0dc-6b50-4591-8f3b-ac39e6d7839e' 2024-09-16T15:14:57.556 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 fd12a0dc-6b50-4591-8f3b-ac39e6d7839e 2024-09-16T15:14:57.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:14:57.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxS+hm2QGCIhAAxotHTYatwlt54mHbHtWTOA== 2024-09-16T15:14:57.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxS+hm2QGCIhAAxotHTYatwlt54mHbHtWTOA=="}' 2024-09-16T15:14:57.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd12a0dc-6b50-4591-8f3b-ac39e6d7839e -i td/divergent-priors/0/new.json 2024-09-16T15:14:57.951 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:14:57.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-16T15:14:57.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxS+hm2QGCIhAAxotHTYatwlt54mHbHtWTOA== --osd-uuid fd12a0dc-6b50-4591-8f3b-ac39e6d7839e 2024-09-16T15:14:58.012 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:58.007+0000 7f6b04b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:14:58.033 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:58.027+0000 7f6b04b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:14:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:58.031+0000 7f6b04b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:14:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:58.031+0000 7f6b04b1d7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:14:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:58.031+0000 7f6b04b1d7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:14:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:58.031+0000 7f6b04b1d7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:14:58.125 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:14:58.119+0000 7f6b04b1d7c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-16T15:15:04.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-16T15:15:04.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:15:04.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:15:04.267 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:15:04.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:15:04.849 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:15:04.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:15:04.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:15:04.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:15:04.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:15:04.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:15:04.882 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:04.879+0000 7f4a9daf87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:15:04.892 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:04.887+0000 7f4a9daf87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:15:04.893 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:04.887+0000 7f4a9daf87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:15:05.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:15:05.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:15:05.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:15:05.324 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:15:05.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:15:05.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:15:05.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:05.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:15:05.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:05.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:05.447 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:05.443+0000 7f4a9daf87c0 -1 Falling back to public interface 2024-09-16T15:15:05.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:06.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:06.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:06.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:15:06.759 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:15:06.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:06.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:07.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:07.948 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:07.943+0000 7f4a9daf87c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:15:08.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:08.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:08.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:15:08.194 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:15:08.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:08.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:08.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:09.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:09.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:09.679 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:15:09.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:15:09.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:09.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:10.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:11.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:11.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:11.152 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:15:11.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:15:11.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:11.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:11.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:12.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:12.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:12.631 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:15:12.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:15:12.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:12.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:13.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:14.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:14.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:14.102 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:15:14.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:15:14.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:14.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:14.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:15.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:15.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:15.609 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:15:15.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:15:15.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:15.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:16.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:17.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:17.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:17.078 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:15:17.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:15:17.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:17.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:17.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:18.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:18.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:18.627 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:15:18.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:15:18.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:18.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:19.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:20.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:20.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:20.068 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:15:20.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:15:20.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:20.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:20.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:21.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:21.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:21.717 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:15:21.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:15:21.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:21.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:22.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:23.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:23.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:23.259 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:15:23.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:15:23.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:23.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:24.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:25.012 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:15:25.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:15:25.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:25.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:26.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:27.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:27.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:27.178 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:15:27.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:15:27.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:27.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:28.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:29.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:29.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:29.274 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:15:29.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:15:29.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:29.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:29.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:30.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:30.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:30.909 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:15:30.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:15:30.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:30.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:15:31.372 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/464580071,v1:127.0.0.1:6803/464580071] [v2:127.0.0.1:6804/464580071,v1:127.0.0.1:6805/464580071] exists,up fd12a0dc-6b50-4591-8f3b-ac39e6d7839e 2024-09-16T15:15:31.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:15:31.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:15:31.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:15:31.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-16T15:15:31.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-09-16T15:15:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:15:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:15:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:15:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:15:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:15:31.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:15:31.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:15:31.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:15:31.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:15:31.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:15:31.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:15:31.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:15:31.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:15:31.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:15:31.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:15:31.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:15:31.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:15:31.382 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:15:31.382 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:15:31.382 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:15:31.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:15:31.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:15:31.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:15:31.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:15:31.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:15:31.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:15:31.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:15:31.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:15:31.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:15:31.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:15:31.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:15:31.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:15:31.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:15:31.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:15:31.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:15:31.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=28dad360-6435-4eca-8975-4d482cf456d8 2024-09-16T15:15:31.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 28dad360-6435-4eca-8975-4d482cf456d8' 2024-09-16T15:15:31.385 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 28dad360-6435-4eca-8975-4d482cf456d8 2024-09-16T15:15:31.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:15:31.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTS+hmQPtLGBAAJXJV80eftLlMMzcVDuTlEQ== 2024-09-16T15:15:31.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTS+hmQPtLGBAAJXJV80eftLlMMzcVDuTlEQ=="}' 2024-09-16T15:15:31.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 28dad360-6435-4eca-8975-4d482cf456d8 -i td/divergent-priors/1/new.json 2024-09-16T15:15:31.943 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:15:31.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-16T15:15:31.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTS+hmQPtLGBAAJXJV80eftLlMMzcVDuTlEQ== --osd-uuid 28dad360-6435-4eca-8975-4d482cf456d8 2024-09-16T15:15:32.003 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:31.999+0000 7f9ddc8ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:15:32.023 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:32.019+0000 7f9ddc8ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:15:32.025 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:32.019+0000 7f9ddc8ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:15:32.025 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:32.023+0000 7f9ddc8ce7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:15:32.025 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:32.023+0000 7f9ddc8ce7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:15:32.025 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:32.023+0000 7f9ddc8ce7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:15:32.138 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:32.135+0000 7f9ddc8ce7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-16T15:15:38.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-16T15:15:38.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:15:38.552 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:15:38.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:15:38.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:15:39.147 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:15:39.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:15:39.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:15:39.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:15:39.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:15:39.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:15:39.190 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:39.187+0000 7f0a1d8f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:15:39.192 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:39.187+0000 7f0a1d8f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:15:39.194 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:39.191+0000 7f0a1d8f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:15:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:15:39.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:15:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:15:39.580 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:15:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:15:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:15:39.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:15:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:39.779 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:39.775+0000 7f0a1d8f57c0 -1 Falling back to public interface 2024-09-16T15:15:40.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:41.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:41.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:41.012 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:15:41.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:15:41.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:41.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:41.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:42.451 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:15:42.447+0000 7f0a1d8f57c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:15:42.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:42.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:42.453 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:15:42.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:15:42.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:42.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:43.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:44.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:44.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:44.170 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:15:44.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:15:44.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:44.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:44.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:45.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:45.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:45.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:15:45.935 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:15:45.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:45.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:46.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:47.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:47.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:47.574 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:15:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:15:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:48.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:49.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:49.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:49.034 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:15:49.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:15:49.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:49.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:49.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:50.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:50.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:50.514 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:15:50.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:15:50.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:50.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:51.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:52.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:52.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:52.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:15:52.042 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:15:52.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:52.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:52.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:53.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:53.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:53.703 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:15:53.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:15:53.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:53.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:54.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:55.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:55.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:55.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:15:55.159 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:15:55.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:55.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:56.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:57.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:57.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:57.225 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:15:57.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:15:57.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:57.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:58.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:15:59.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:15:59.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:15:59.060 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:15:59.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:15:59.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:15:59.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:15:59.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:00.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:00.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:00.906 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:16:00.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:16:00.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:00.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:16:01.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:02.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:02.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:02.686 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:16:02.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:16:02.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:02.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:16:03.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:04.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:04.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:04.750 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:16:04.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:16:04.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:04.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:16:05.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:06.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:06.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:06.798 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:16:06.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:16:06.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:06.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:16:07.280 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/723941755,v1:127.0.0.1:6811/723941755] [v2:127.0.0.1:6812/723941755,v1:127.0.0.1:6813/723941755] exists,up 28dad360-6435-4eca-8975-4d482cf456d8 2024-09-16T15:16:07.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:16:07.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:16:07.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:16:07.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-16T15:16:07.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-09-16T15:16:07.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:16:07.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:16:07.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T15:16:07.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:16:07.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-16T15:16:07.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:16:07.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:16:07.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:16:07.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:16:07.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:16:07.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:16:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-16T15:16:07.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:16:07.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e9bbafa-8426-4f88-a475-b4a41f336294 2024-09-16T15:16:07.289 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 9e9bbafa-8426-4f88-a475-b4a41f336294 2024-09-16T15:16:07.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9e9bbafa-8426-4f88-a475-b4a41f336294' 2024-09-16T15:16:07.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:16:07.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3S+hmQsDaEhAAGY/s5q7v5+m+20R5DrDDgg== 2024-09-16T15:16:07.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3S+hmQsDaEhAAGY/s5q7v5+m+20R5DrDDgg=="}' 2024-09-16T15:16:07.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e9bbafa-8426-4f88-a475-b4a41f336294 -i td/divergent-priors/2/new.json 2024-09-16T15:16:07.954 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:16:07.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-16T15:16:07.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3S+hmQsDaEhAAGY/s5q7v5+m+20R5DrDDgg== --osd-uuid 9e9bbafa-8426-4f88-a475-b4a41f336294 2024-09-16T15:16:08.019 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:08.019+0000 7ff3e193f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:16:08.035 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:08.035+0000 7ff3e193f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:16:08.037 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:08.039+0000 7ff3e193f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:16:08.037 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:08.039+0000 7ff3e193f7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:16:08.037 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:08.039+0000 7ff3e193f7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:16:08.038 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:08.039+0000 7ff3e193f7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:16:08.127 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:08.127+0000 7ff3e193f7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-16T15:16:14.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-16T15:16:14.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:16:14.286 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T15:16:14.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T15:16:14.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:16:14.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T15:16:14.907 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:16:14.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:16:14.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:16:14.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:16:14.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:16:14.948 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:14.946+0000 7f951a68d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:16:14.960 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:14.958+0000 7f951a68d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:16:14.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:14.962+0000 7f951a68d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:16:15.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T15:16:15.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:16:15.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:16:15.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:16:15.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:16:15.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:15.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:16:15.376 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:16:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:15.538 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:15.538+0000 7f951a68d7c0 -1 Falling back to public interface 2024-09-16T15:16:15.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:16.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:16.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:16:16.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:16:16.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:16.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:17.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:18.123 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:16:18.122+0000 7f951a68d7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:16:18.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:18.298 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:16:18.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:18.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:16:18.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:18.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:18.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:19.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:19.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:19.961 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:16:19.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:16:19.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:19.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:20.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:21.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:21.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:21.441 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:16:21.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:16:21.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:21.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:22.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:23.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:23.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:23.053 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:16:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:16:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:23.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:24.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:24.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:24.578 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:16:24.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:16:24.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:24.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:25.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:26.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:26.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:26.288 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:16:26.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:16:26.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:26.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:26.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:27.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:27.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:27.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:16:27.910 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:16:27.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:27.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:29.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:29.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:29.368 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:16:29.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:16:29.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:29.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:29.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:30.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:30.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:30.845 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:16:30.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:16:30.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:30.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:31.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:32.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:32.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:32.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:16:32.295 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:16:32.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:32.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:33.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:34.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:34.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:34.716 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:16:34.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:16:34.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:34.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:35.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:36.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:36.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:36.575 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:16:36.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:16:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:38.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:39.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:39.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:39.008 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:16:39.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:16:39.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:39.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:39.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:40.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:40.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:40.583 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:16:40.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:16:40.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:40.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:41.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:16:42.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:16:42.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:16:42.510 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:16:42.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:16:42.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:16:42.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:16:43.018 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/990712287,v1:127.0.0.1:6819/990712287] [v2:127.0.0.1:6820/990712287,v1:127.0.0.1:6821/990712287] exists,up 9e9bbafa-8426-4f88-a475-b4a41f336294 2024-09-16T15:16:43.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:16:43.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:16:43.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:16:43.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-09-16T15:16:43.777 INFO:tasks.workunit.client.0.smithi039.stderr:noout is set 2024-09-16T15:16:43.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-09-16T15:16:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:noin is set 2024-09-16T15:16:44.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-09-16T15:16:45.610 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T15:16:45.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-09-16T15:16:45.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-16T15:16:46.286 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T15:16:46.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:16:47.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-09-16T15:16:48.239 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 3 2024-09-16T15:16:48.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-09-16T15:16:49.202 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 2 2024-09-16T15:16:49.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-09-16T15:16:49.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:16:49.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:16:49.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:16:49.714 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:16:49.714 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:16:49.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:16:49.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:16:49.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:16:49.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836502 2024-09-16T15:16:49.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836502 2024-09-16T15:16:49.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836502' 2024-09-16T15:16:49.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:16:49.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:16:50.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672980 2024-09-16T15:16:50.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672980 2024-09-16T15:16:50.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836502 1-42949672980' 2024-09-16T15:16:50.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:16:50.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:16:50.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509450 2024-09-16T15:16:50.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509450 2024-09-16T15:16:50.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-64424509450' 2024-09-16T15:16:50.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:16:50.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836502 2024-09-16T15:16:50.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:16:50.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:16:50.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836502 2024-09-16T15:16:50.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:16:50.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836502 2024-09-16T15:16:50.260 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836502 2024-09-16T15:16:50.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-16T15:16:50.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:16:50.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-16T15:16:50.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:16:50.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672980 2024-09-16T15:16:50.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:16:50.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:16:50.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672980 2024-09-16T15:16:50.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:16:50.691 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672980 2024-09-16T15:16:50.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672980 2024-09-16T15:16:50.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-16T15:16:50.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:16:51.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-16T15:16:51.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:16:51.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509450 2024-09-16T15:16:51.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:16:51.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:16:51.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509450 2024-09-16T15:16:51.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:16:51.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509450 2024-09-16T15:16:51.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-16T15:16:51.170 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509450 2024-09-16T15:16:51.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:16:51.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-16T15:16:51.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-09-16T15:16:51.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:16:51.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:16:51.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:16:51.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:16:51.637 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:16:51.637 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:16:51.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:16:51.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:16:51.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:16:51.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:16:51.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:16:51.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:16:51.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:16:51.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:16:51.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:16:52.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:16:52.159 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:16:52.159 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:16:52.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:16:52.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:16:52.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:16:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836505 2024-09-16T15:16:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836505 2024-09-16T15:16:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505' 2024-09-16T15:16:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:16:52.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:16:52.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672983 2024-09-16T15:16:52.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672983 2024-09-16T15:16:52.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672983' 2024-09-16T15:16:52.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:16:52.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:16:52.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509452 2024-09-16T15:16:52.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509452 2024-09-16T15:16:52.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509452' 2024-09-16T15:16:52.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:16:52.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836505 2024-09-16T15:16:52.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:16:52.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:16:52.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836505 2024-09-16T15:16:52.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:16:52.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836505 2024-09-16T15:16:52.707 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836505 2024-09-16T15:16:52.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-16T15:16:52.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:16:53.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836503 -lt 21474836505 2024-09-16T15:16:53.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:16:54.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:16:54.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:16:54.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-16T15:16:54.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:16:54.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:16:54.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672983 2024-09-16T15:16:54.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:16:54.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:16:54.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672983 2024-09-16T15:16:54.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672983 2024-09-16T15:16:54.637 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672983 2024-09-16T15:16:54.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-16T15:16:54.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:16:55.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-16T15:16:55.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:16:55.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:16:55.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509452 2024-09-16T15:16:55.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:16:55.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509452 2024-09-16T15:16:55.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:16:55.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509452 2024-09-16T15:16:55.118 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509452 2024-09-16T15:16:55.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-16T15:16:55.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:16:55.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509453 -lt 64424509452 2024-09-16T15:16:55.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:16:55.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:16:55.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:16:56.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:16:56.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:16:56.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:16:56.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:16:56.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:16:56.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:16:56.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:16:56.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:16:56.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T15:16:56.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:16:56.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:16:56.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:16:57.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T15:16:57.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:16:57.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:16:57.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-09-16T15:16:57.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-09-16T15:16:57.498 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:16:57.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-09-16T15:16:57.514 INFO:tasks.workunit.client.0.smithi039.stdout:primary and soon to be divergent is 1 2024-09-16T15:16:57.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-09-16T15:16:57.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:16:57.921 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:16:57.921 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T15:16:47.426245+0000 0'0 28:15 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T21:23:20.577915+0000 0 0 2024-09-16T15:16:57.921 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:16:57.921 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:16:57.922 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:16:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-09-16T15:16:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-16T15:16:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-09-16T15:16:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-09-16T15:16:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-16T15:16:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-09-16T15:16:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-09-16T15:16:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-16T15:16:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-09-16T15:16:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-09-16T15:16:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-09-16T15:16:57.938 INFO:tasks.workunit.client.0.smithi039.stdout:writing initial objects 2024-09-16T15:16:57.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-09-16T15:16:57.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:57.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-09-16T15:16:58.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:58.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-09-16T15:16:58.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:58.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-09-16T15:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-09-16T15:16:58.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:58.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-09-16T15:16:58.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:58.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-09-16T15:16:58.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:58.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-09-16T15:16:59.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:59.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-09-16T15:16:59.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:59.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-09-16T15:16:59.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:59.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-09-16T15:16:59.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:59.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-09-16T15:16:59.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:16:59.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-09-16T15:17:00.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:00.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-09-16T15:17:00.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:00.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-09-16T15:17:00.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:00.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-09-16T15:17:00.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:00.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-09-16T15:17:00.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:00.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-09-16T15:17:01.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:01.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-09-16T15:17:01.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:01.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-09-16T15:17:01.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:01.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-09-16T15:17:01.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:01.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-09-16T15:17:01.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:01.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-09-16T15:17:01.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:01.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-09-16T15:17:02.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:02.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-09-16T15:17:02.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:02.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-09-16T15:17:02.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:02.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-09-16T15:17:02.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:02.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-09-16T15:17:02.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:02.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-09-16T15:17:02.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:02.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-09-16T15:17:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:03.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-09-16T15:17:03.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:03.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-09-16T15:17:03.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:03.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-09-16T15:17:03.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:03.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-09-16T15:17:03.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:03.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-09-16T15:17:03.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:03.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-09-16T15:17:04.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:04.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-09-16T15:17:04.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:04.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-09-16T15:17:04.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:04.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-09-16T15:17:04.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:04.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-09-16T15:17:04.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:04.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-09-16T15:17:05.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:05.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-09-16T15:17:05.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:05.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-09-16T15:17:05.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:05.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-09-16T15:17:05.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:05.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-09-16T15:17:05.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:05.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-09-16T15:17:05.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:05.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-09-16T15:17:06.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:06.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-09-16T15:17:06.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:06.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-09-16T15:17:06.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:06.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-09-16T15:17:06.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:06.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-09-16T15:17:06.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:06.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-09-16T15:17:07.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:07.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-09-16T15:17:07.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:07.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-09-16T15:17:07.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:07.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-09-16T15:17:07.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:07.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-09-16T15:17:07.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:07.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-09-16T15:17:08.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:08.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-09-16T15:17:08.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:08.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-09-16T15:17:08.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:08.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-09-16T15:17:08.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:08.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-09-16T15:17:08.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:08.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-09-16T15:17:08.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:08.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-09-16T15:17:09.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:09.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-09-16T15:17:09.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:09.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-09-16T15:17:09.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:09.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-09-16T15:17:09.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:09.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-09-16T15:17:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-09-16T15:17:10.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:10.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-09-16T15:17:10.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:10.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-09-16T15:17:10.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:10.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-09-16T15:17:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:10.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-09-16T15:17:10.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:10.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-09-16T15:17:10.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:10.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-09-16T15:17:11.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:11.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-09-16T15:17:11.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:11.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-09-16T15:17:11.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:11.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-09-16T15:17:11.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:11.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-09-16T15:17:11.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:11.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-09-16T15:17:11.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:11.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-09-16T15:17:12.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:12.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-09-16T15:17:12.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:12.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-09-16T15:17:12.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:12.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-09-16T15:17:12.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:12.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-09-16T15:17:12.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:12.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-09-16T15:17:12.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:12.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-09-16T15:17:13.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:13.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-09-16T15:17:13.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:13.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-09-16T15:17:13.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:13.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-09-16T15:17:13.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:13.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-09-16T15:17:13.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:13.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-09-16T15:17:14.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:14.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-09-16T15:17:14.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:14.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-09-16T15:17:14.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:14.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-09-16T15:17:14.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:14.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-09-16T15:17:14.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:14.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-09-16T15:17:15.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:15.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-09-16T15:17:15.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:15.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-09-16T15:17:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-09-16T15:17:15.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:15.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-09-16T15:17:15.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-16T15:17:15.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-09-16T15:17:15.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T15:17:15.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-09-16T15:17:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:17:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:17:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:17:15.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T15:17:15.891 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:17:15.891 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:17:15.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:17:15.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:17:15.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:17:15.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T15:17:15.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:17:15.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:17:15.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:17:15.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:17:15.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:17:16.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:17:16.474 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:17:16.474 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:17:16.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:17:16.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:17:16.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:17:16.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836512 2024-09-16T15:17:16.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836512 2024-09-16T15:17:16.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836512' 2024-09-16T15:17:16.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:17:16.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:17:16.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672989 2024-09-16T15:17:16.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672989 2024-09-16T15:17:16.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836512 1-42949672989' 2024-09-16T15:17:16.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:17:16.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:17:16.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509459 2024-09-16T15:17:16.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509459 2024-09-16T15:17:16.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836512 1-42949672989 2-64424509459' 2024-09-16T15:17:16.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:17:16.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836512 2024-09-16T15:17:16.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:17:16.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:17:16.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836512 2024-09-16T15:17:16.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:17:16.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836512 2024-09-16T15:17:16.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-16T15:17:16.998 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836512 2024-09-16T15:17:16.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:17:17.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836509 -lt 21474836512 2024-09-16T15:17:17.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:17:18.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:17:18.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:17:18.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-16T15:17:18.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:17:18.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672989 2024-09-16T15:17:18.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:17:18.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:17:18.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672989 2024-09-16T15:17:18.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:17:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672989 2024-09-16T15:17:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-16T15:17:18.927 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672989 2024-09-16T15:17:18.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:17:19.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672990 -lt 42949672989 2024-09-16T15:17:19.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:17:19.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509459 2024-09-16T15:17:19.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:17:19.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:17:19.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509459 2024-09-16T15:17:19.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:17:19.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509459 2024-09-16T15:17:19.354 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509459 2024-09-16T15:17:19.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-16T15:17:19.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:17:19.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-16T15:17:19.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:17:19.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:17:19.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:17:20.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:17:20.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:17:20.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:17:20.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:17:20.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:17:20.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:17:20.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:17:20.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:17:20.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T15:17:20.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:17:20.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:17:20.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:17:21.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T15:17:21.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:17:21.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:17:21.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-09-16T15:17:21.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-16T15:17:21.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-16T15:17:21.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-16T15:17:21.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T15:17:21.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-09-16T15:17:21.841 INFO:tasks.workunit.client.0.smithi039.stdout:blackholing osds 0 2 2024-09-16T15:17:21.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-09-16T15:17:21.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:17:22.261 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:17:22.261 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-16T15:16:47.426245+0000 29'100 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T21:23:20.577915+0000 0 0 2024-09-16T15:17:22.261 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:17:22.261 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:17:22.261 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:17:22.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-16T15:17:22.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-09-16T15:17:22.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T15:17:22.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T15:17:22.277 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:17:22.277 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:17:22.278 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:17:22.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.0.asok 2024-09-16T15:17:22.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-16T15:17:22.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-16T15:17:22.380 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:17:22.380 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-16T15:17:22.380 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:17:22.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-16T15:17:22.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-09-16T15:17:22.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T15:17:22.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T15:17:22.395 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:17:22.395 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:17:22.395 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:17:22.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.2.asok 2024-09-16T15:17:22.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-16T15:17:22.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-16T15:17:22.508 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:17:22.509 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-16T15:17:22.509 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:17:22.521 INFO:tasks.workunit.client.0.smithi039.stdout:create new divergent objects 2024-09-16T15:17:22.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-09-16T15:17:22.522 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-09-16T15:17:22.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-16T15:17:22.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-16T15:17:22.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-09-16T15:17:22.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-09-16T15:17:22.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-09-16T15:17:22.525 INFO:tasks.workunit.client.0.smithi039.stdout:writing divergent objects 2024-09-16T15:17:22.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-09-16T15:17:22.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-16T15:17:22.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-16T15:17:22.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-09-16T15:17:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-09-16T15:17:22.527 INFO:tasks.workunit.client.0.smithi039.stdout:remove divergent objects 2024-09-16T15:17:22.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-09-16T15:17:22.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-09-16T15:17:22.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-16T15:17:22.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-09-16T15:17:22.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-09-16T15:17:22.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-16T15:17:22.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-09-16T15:17:22.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-09-16T15:17:22.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-09-16T15:17:22.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-09-16T15:17:22.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-09-16T15:17:32.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-09-16T15:17:32.540 INFO:tasks.workunit.client.0.smithi039.stderr:rados: no process found 2024-09-16T15:17:32.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-09-16T15:17:32.540 INFO:tasks.workunit.client.0.smithi039.stdout:killing all the osds 2024-09-16T15:17:32.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:17:33.023 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:17:33.023 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 active+clean 2024-09-16T15:16:47.426245+0000 29'106 29:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T21:23:20.577915+0000 0 0 2024-09-16T15:17:33.024 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:17:33.024 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:17:33.024 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:17:33.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-09-16T15:17:33.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:17:33.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:17:33.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:17:33.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:17:33.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:17:33.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:17:33.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-16T15:17:33.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-09-16T15:17:33.714 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T15:17:33.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-16T15:17:33.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-09-16T15:17:34.292 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-16T15:17:34.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-16T15:17:34.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-09-16T15:17:34.950 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-16T15:17:34.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-16T15:17:34.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-09-16T15:17:35.565 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T15:17:35.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-16T15:17:35.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-09-16T15:17:36.152 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already out. 2024-09-16T15:17:36.168 INFO:tasks.workunit.client.0.smithi039.stdout:bringing up non_divergent 0 2 2024-09-16T15:17:36.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-09-16T15:17:36.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:17:36.554 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:17:36.554 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 stale+active+clean 2024-09-16T15:16:47.426245+0000 29'106 29:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T21:23:20.577915+0000 0 0 2024-09-16T15:17:36.554 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:17:36.554 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:17:36.554 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:17:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-16T15:17:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-09-16T15:17:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:17:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:17:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:17:36.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:17:36.573 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:17:36.573 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:17:36.573 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:17:36.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:17:36.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:17:36.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:17:36.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:17:36.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:17:36.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:17:36.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:17:36.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:17:36.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:17:36.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:17:36.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:17:36.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-16T15:17:36.575 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:17:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:17:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:17:36.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-16T15:17:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:17:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:17:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:17:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:17:36.614 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:17:36.613+0000 7fdd49beb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:17:36.617 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:17:36.617+0000 7fdd49beb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:17:36.618 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:17:36.617+0000 7fdd49beb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:17:37.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:17:37.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:17:37.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:17:37.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:17:37.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:17:37.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:17:37.068 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:17:37.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:17:37.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:17:37.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:17:37.182 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:17:37.181+0000 7fdd49beb7c0 -1 Falling back to public interface 2024-09-16T15:17:37.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:17:38.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:17:38.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:17:38.513 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:17:38.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:17:38.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:17:38.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:17:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:17:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:17:39.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:17:39.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:17:39.976 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:17:39.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:17:39.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:17:40.144 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:17:40.141+0000 7fdd49beb7c0 -1 osd.0 29 log_to_monitors true 2024-09-16T15:17:40.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:17:41.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:17:41.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:17:41.431 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:17:41.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:17:41.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:17:41.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:17:41.857 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1920211368,v1:127.0.0.1:6803/1920211368] [v2:127.0.0.1:6804/1920211368,v1:127.0.0.1:6805/1920211368] exists,up fd12a0dc-6b50-4591-8f3b-ac39e6d7839e 2024-09-16T15:17:41.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:17:41.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:17:41.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:17:41.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-16T15:17:41.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:17:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:17:41.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:17:41.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:17:41.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:17:41.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:17:41.862 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:17:41.862 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:17:41.862 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:17:41.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:17:41.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:17:41.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:17:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:17:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:17:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:17:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:17:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:17:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:17:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:17:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:17:41.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-16T15:17:41.863 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:17:41.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T15:17:41.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:17:41.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-16T15:17:41.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T15:17:41.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:17:41.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:17:41.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:17:41.896 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:17:41.897+0000 7f97afb8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:17:41.899 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:17:41.897+0000 7f97afb8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:17:41.900 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:17:41.901+0000 7f97afb8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:17:42.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T15:17:42.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:17:42.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:17:42.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:17:42.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:17:42.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:17:42.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:17:42.356 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:17:42.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:17:42.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:17:42.443 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:17:42.441+0000 7f97afb8d7c0 -1 Falling back to public interface 2024-09-16T15:17:42.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:17:43.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:17:43.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:17:43.900 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:17:43.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:17:43.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:17:43.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:17:44.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:17:45.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:17:45.368 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:17:45.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:17:45.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:17:45.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:17:45.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:17:45.517 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:17:45.517+0000 7f97afb8d7c0 -1 osd.2 29 log_to_monitors true 2024-09-16T15:17:45.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:17:46.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:17:46.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:17:46.839 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:17:46.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:17:46.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:17:46.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:17:47.290 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/1944839882,v1:127.0.0.1:6811/1944839882] [v2:127.0.0.1:6812/1944839882,v1:127.0.0.1:6813/1944839882] exists,up 9e9bbafa-8426-4f88-a475-b4a41f336294 2024-09-16T15:17:47.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:17:47.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:17:47.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:17:47.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-16T15:17:47.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-09-16T15:17:47.900 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already in. 2024-09-16T15:17:47.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-16T15:17:47.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-09-16T15:17:48.609 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already in. 2024-09-16T15:17:48.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T15:17:48.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-09-16T15:17:48.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:17:48.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:17:48.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:17:48.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T15:17:48.634 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:17:48.634 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:17:48.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:17:48.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:17:48.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:17:48.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T15:17:48.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:17:48.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:17:48.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:17:48.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:17:48.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:17:49.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:17:49.215 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:17:49.215 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:17:49.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:17:49.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:17:49.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:17:49.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=154618822660 2024-09-16T15:17:49.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 154618822660 2024-09-16T15:17:49.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822660' 2024-09-16T15:17:49.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:17:49.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:17:49.511 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T15:17:49.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T15:17:49.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T15:17:49.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T15:17:49.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:17:49.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:17:49.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=163208757251 2024-09-16T15:17:49.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 163208757251 2024-09-16T15:17:49.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-16T15:17:49.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:17:49.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-154618822660 2024-09-16T15:17:49.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:17:49.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:17:49.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-154618822660 2024-09-16T15:17:49.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:17:49.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=154618822660 2024-09-16T15:17:49.684 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 154618822660 2024-09-16T15:17:49.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-16T15:17:49.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:17:50.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 154618822658 -lt 154618822660 2024-09-16T15:17:50.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:17:51.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:17:51.164 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:17:51.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-16T15:17:51.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:17:51.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:17:51.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-163208757251 2024-09-16T15:17:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:17:51.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-163208757251 2024-09-16T15:17:51.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:17:51.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=163208757251 2024-09-16T15:17:51.649 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 163208757251 2024-09-16T15:17:51.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-16T15:17:51.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:17:52.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-16T15:17:52.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:17:52.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:17:52.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:17:52.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:17:52.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:17:52.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:17:52.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:17:52.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:17:52.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:17:52.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:17:52.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:17:53.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:17:53.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:17:53.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:17:53.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:17:53.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:17:53.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T15:17:53.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T15:17:53.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T15:17:53.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:17:53.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T15:17:53.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:17:53.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:17:53.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:17:53.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:17:53.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:17:53.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:17:53.701 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:17:53.701 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:17:54.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:17:54.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:17:54.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:17:54.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:17:54.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:17:54.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:17:54.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:17:54.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:17:54.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:17:54.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:17:54.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:17:54.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:17:54.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:17:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:17:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:17:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-16T15:17:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:17:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T15:17:55.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:17:55.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:17:55.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:17:55.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:17:55.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:17:55.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:17:55.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:17:55.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:17:55.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:17:55.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:17:55.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:17:55.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:17:56.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:17:56.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:17:56.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:17:56.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:17:56.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:17:56.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:17:56.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:17:56.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:17:56.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:17:56.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:17:56.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:17:56.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-16T15:17:56.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:17:56.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T15:17:57.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:17:57.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:17:57.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:17:57.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:17:57.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:17:57.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:17:57.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:17:57.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:17:57.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:17:57.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:17:57.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:17:57.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:17:58.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:17:58.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:17:58.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:17:58.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:17:58.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:17:58.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:17:58.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:17:58.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:17:58.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:17:58.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:17:58.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:17:58.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-16T15:17:58.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:17:58.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-16T15:17:59.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:17:59.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:17:59.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:17:59.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:17:59.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:17:59.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:17:59.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:17:59.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:00.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:00.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:00.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:00.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:00.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:00.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:00.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:00.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:00.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:00.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:00.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:00.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:00.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:01.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:01.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:01.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-16T15:18:01.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:01.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-16T15:18:02.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:18:02.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:02.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:02.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:02.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:02.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:02.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:02.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:03.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:03.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:03.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:03.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:03.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:03.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:03.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:03.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:03.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:03.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:03.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:03.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:03.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-16T15:18:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-16T15:18:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:18:07.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:07.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:07.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:07.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:07.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:07.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:07.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:08.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:08.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:08.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:08.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:08.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:08.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:08.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:08.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:08.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:08.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:08.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:08.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:08.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:09.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:09.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:09.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-16T15:18:09.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:09.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-16T15:18:15.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:18:15.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:15.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:15.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:15.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:15.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:15.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:15.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:16.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:16.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:16.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:16.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:16.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:16.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:16.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:16.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:16.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:16.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:16.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:16.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:16.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:17.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:17.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:17.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-16T15:18:17.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:17.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-16T15:18:24.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:18:24.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:24.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:24.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:24.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:24.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:24.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:24.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:24.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:24.889 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:24.889 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:24.889 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:25.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:25.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:25.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:25.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:25.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:25.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:25.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:25.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:25.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:25.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:25.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:25.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-16T15:18:25.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T15:18:26.525 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:18:26.525 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "d4fe5bcf-08b5-4ddc-8f6f-b23fc2932725", 2024-09-16T15:18:26.525 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "18.2.4-845-ge4e227b3", 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "e4e227b37858f5109138494e01e5ba1ba7b8ffe8", 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-16T15:18:26.526968+0000", 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-16T15:18:26.526 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:18:26.527 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-16T15:18:26.528 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.529 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:14:55.778774Z", 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:14:55.778774Z", 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 18, 2024-09-16T15:18:26.530 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "reef", 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-16T15:18:26.531 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: "reef" 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.532 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-16T15:18:26.533 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 41, 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:14:56.721200+0000", 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:17:48.611673+0000", 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-16T15:17:45.794496+0000", 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-16T15:17:48.285307+0000", 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-16T15:18:26.534 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 9, 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "reef", 2024-09-16T15:18:26.535 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-16T15:16:45.972443+0000", 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T15:18:26.536 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 1, 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 1, 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 1, 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 1, 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.537 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "29", 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-16T15:18:26.538 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-16T15:18:26.539 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 2.9900000095367432, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 2.9900000095367432, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 0.67000001668930054, 2024-09-16T15:18:26.540 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 2, 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 2, 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "fd12a0dc-6b50-4591-8f3b-ac39e6d7839e", 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:18:26.541 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 36, 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 40, 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 30, 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.542 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-16T15:18:26.543 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.544 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.545 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/1920211368", 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/1920211368", 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1920211368", 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1920211368", 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "28dad360-6435-4eca-8975-4d482cf456d8", 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-16T15:18:26.546 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 23, 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 31, 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:18:26.547 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.548 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.549 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/723941755", 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/723941755", 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/723941755", 2024-09-16T15:18:26.550 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/723941755", 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "9e9bbafa-8426-4f88-a475-b4a41f336294", 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 38, 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 0, 2024-09-16T15:18:26.551 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 32, 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:18:26.552 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.553 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:18:26.554 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/1944839882", 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/1944839882", 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1944839882", 2024-09-16T15:18:26.555 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1944839882", 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:17:33.503177+0000", 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:15:30.624524+0000", 2024-09-16T15:18:26.556 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:17:34.101398+0000", 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:16:06.154172+0000", 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:17:34.743868+0000", 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:18:26.557 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:18:26.558 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:18:26.558 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:16:42.396512+0000", 2024-09-16T15:18:26.558 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:18:26.558 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.558 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.558 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-16T15:18:26.558 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-16T15:18:26.558 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-16T15:18:26.558 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-16T15:18:26.558 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-16T15:18:26.559 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/1920211368,v1:127.0.0.1:6805/1920211368]", 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:18:26.560 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:18:26.561 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:15:03.606999Z", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:18:26.562 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/1920211368,v1:127.0.0.1:6803/1920211368]", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1920211368,v1:127.0.0.1:6809/1920211368]", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1920211368,v1:127.0.0.1:6807/1920211368]", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:18:26.563 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/723941755,v1:127.0.0.1:6813/723941755]", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:18:26.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:15:37.864050Z", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:18:26.565 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/723941755,v1:127.0.0.1:6811/723941755]", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/723941755,v1:127.0.0.1:6817/723941755]", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/723941755,v1:127.0.0.1:6815/723941755]", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:18:26.566 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/1944839882,v1:127.0.0.1:6813/1944839882]", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:18:26.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:18:26.568 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:16:13.608965Z", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/1944839882,v1:127.0.0.1:6811/1944839882]", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1944839882,v1:127.0.0.1:6817/1944839882]", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1944839882,v1:127.0.0.1:6815/1944839882]", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-16T15:18:26.570 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [] 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 36 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 38 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.571 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-16T15:18:26.572 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 41, 2024-09-16T15:18:26.572 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-16T15:18:26.572 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-16T15:18:26.572 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.572 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:18:26.572 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-16T15:18:26.572 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:18:26.572 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.572 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.572 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:18:26.572 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-16T15:18:26.573 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-16T15:18:26.574 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-16T15:18:26.575 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.576 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.577 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.578 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:18:26.579 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.580 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:18:26.581 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.582 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-16T15:18:26.583 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-16T15:18:26.584 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 0, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.585 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T15:18:26.586 INFO:tasks.workunit.client.0.smithi039.stderr:report 593337772 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2" 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.587 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 1, 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-16T15:18:26.588 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 61700, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 100, 2024-09-16T15:18:26.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 100, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:18:26.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:18:26.591 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:18:26.591 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:18:26.591 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:18:26.591 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:18:26.591 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:18:26.591 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:18:26.591 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:18:26.591 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:18:26.591 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.591 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:18:26.591 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 16, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 16, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-16T15:18:26.592 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83624, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2104, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489176, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:18:26.593 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036916224, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2154496, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 459731, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:18:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-16T15:18:26.595 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83624, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2104, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489176, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036916224, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2154496, 2024-09-16T15:18:26.596 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 459731, 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:18:26.597 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 61700, 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T15:18:26.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 100, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 100, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:18:26.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:18:26.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1220608, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 184566, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:18:26.601 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 16, 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 16, 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 154618822667 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672993 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 163208757258 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 1 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:18:26.603 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.604 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 5, 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:17:36.306141+0000", 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 94, 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 370, 2024-09-16T15:18:26.605 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-16T15:18:26.606 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:18:26.606 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 510, 2024-09-16T15:18:26.606 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-16T15:18:26.606 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-16T15:18:26.606 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:18:26.606 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:18:26.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T15:18:26.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-09-16T15:18:26.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-09-16T15:18:26.607 INFO:tasks.workunit.client.0.smithi039.stdout:writing non-divergent object existing_4 2024-09-16T15:18:26.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-09-16T15:18:26.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:18:26.950 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:18:26.951 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-16T15:17:48.921357+0000 29'100 41:124 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:08:45.081625+0000 0 0 2024-09-16T15:18:26.951 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:18:26.951 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:18:26.951 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:18:26.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-09-16T15:18:27.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T15:18:27.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-09-16T15:18:27.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:18:27.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:18:27.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:18:27.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T15:18:27.079 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:18:27.079 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:18:27.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:18:27.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:18:27.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:18:27.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T15:18:27.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:18:27.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:18:27.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:18:27.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:18:27.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:18:27.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:18:27.594 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:18:27.594 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:18:27.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:18:27.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:18:27.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:18:27.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=154618822670 2024-09-16T15:18:27.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 154618822670 2024-09-16T15:18:27.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822670' 2024-09-16T15:18:27.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:18:27.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:18:27.894 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T15:18:27.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T15:18:27.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T15:18:27.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T15:18:27.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:18:27.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:18:28.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=163208757261 2024-09-16T15:18:28.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 163208757261 2024-09-16T15:18:28.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822670 2-163208757261' 2024-09-16T15:18:28.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:18:28.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-154618822670 2024-09-16T15:18:28.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:18:28.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:18:28.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-154618822670 2024-09-16T15:18:28.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:18:28.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=154618822670 2024-09-16T15:18:28.067 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 154618822670 2024-09-16T15:18:28.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 154618822670' 2024-09-16T15:18:28.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:18:28.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 154618822670 -lt 154618822670 2024-09-16T15:18:28.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:18:28.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-163208757261 2024-09-16T15:18:28.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:18:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:18:28.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-163208757261 2024-09-16T15:18:28.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:18:28.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=163208757261 2024-09-16T15:18:28.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 163208757261' 2024-09-16T15:18:28.491 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 163208757261 2024-09-16T15:18:28.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:18:28.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 163208757261 -lt 163208757261 2024-09-16T15:18:28.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:18:28.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:28.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:29.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:18:29.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:29.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:29.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:29.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:29.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:29.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:29.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:29.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:29.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:29.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:29.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:30.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:30.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T15:18:30.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T15:18:30.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T15:18:30.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:30.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T15:18:30.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:18:30.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:30.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:30.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:30.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:30.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:30.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:30.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:31.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:31.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:31.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:31.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:31.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:31.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:31.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:31.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:31.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:31.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:31.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:31.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:31.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:32.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:32.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:32.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-16T15:18:32.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:32.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T15:18:32.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:18:32.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:32.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:32.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:32.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:32.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:32.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:32.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:32.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:32.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:32.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:32.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:33.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:33.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:33.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:33.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:33.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:33.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:33.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:33.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:33.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:33.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:33.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:33.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-16T15:18:33.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:33.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T15:18:34.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:18:34.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:34.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:34.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:34.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:34.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:34.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:34.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:34.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:34.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:34.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:34.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:35.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:35.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:35.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:35.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:35.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:35.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:35.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:35.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:35.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:35.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:35.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:35.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-16T15:18:35.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:35.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-16T15:18:36.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:18:36.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:36.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:36.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:36.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:36.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:36.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:36.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:37.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:37.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:37.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:37.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:37.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:37.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:37.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:37.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:37.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:37.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:37.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:37.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:37.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:38.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:38.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:38.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-16T15:18:38.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:38.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-16T15:18:39.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:18:39.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:39.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:39.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:39.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:39.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:39.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:39.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:40.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:40.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:40.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:40.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:40.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:40.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:40.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:40.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:40.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:40.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:40.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:40.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:40.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:41.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:41.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:41.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-16T15:18:41.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:41.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-16T15:18:44.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:18:44.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:44.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:44.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:44.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:44.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:44.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:44.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:45.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:45.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:45.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:45.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:45.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:45.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:45.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:45.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:45.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:45.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:45.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:45.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:45.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:46.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:46.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:46.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-16T15:18:46.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:46.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-16T15:18:52.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:18:52.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:18:52.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:18:52.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:18:52.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:18:52.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:18:52.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:18:52.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:18:53.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:18:53.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:18:53.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:18:53.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:18:53.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:18:53.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:18:53.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:18:53.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:18:53.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:18:53.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:18:53.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:18:53.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:18:53.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:18:54.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:18:54.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:18:54.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-16T15:18:54.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:18:54.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-16T15:19:01.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:19:01.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:19:01.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:19:01.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:19:01.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:19:01.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:19:01.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:19:01.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:19:01.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:19:01.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:19:01.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:19:01.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:19:02.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:19:02.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:19:02.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:19:02.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:19:02.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:19:02.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:19:02.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:19:02.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:19:02.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:19:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:19:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:19:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-16T15:19:03.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "d4fe5bcf-08b5-4ddc-8f6f-b23fc2932725", 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "18.2.4-845-ge4e227b3", 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "e4e227b37858f5109138494e01e5ba1ba7b8ffe8", 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-16T15:19:03.569175+0000", 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:19:03.568 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.569 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 2 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.570 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-16T15:19:03.571 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:14:55.778774Z", 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:14:55.778774Z", 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 18, 2024-09-16T15:19:03.572 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "reef", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-16T15:19:03.573 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: "reef" 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-16T15:19:03.574 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 41, 2024-09-16T15:19:03.575 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:14:56.721200+0000", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:17:48.611673+0000", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-16T15:17:45.794496+0000", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-16T15:17:48.285307+0000", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-16T15:19:03.576 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 9, 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "reef", 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.577 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-16T15:16:45.972443+0000", 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 1, 2024-09-16T15:19:03.578 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 1, 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 1, 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 1, 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "29", 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T15:19:03.579 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-16T15:19:03.580 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-16T15:19:03.581 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 2.9900000095367432, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 2.9900000095367432, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 0.67000001668930054, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 2, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 2, 2024-09-16T15:19:03.582 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "fd12a0dc-6b50-4591-8f3b-ac39e6d7839e", 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:19:03.583 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 36, 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 40, 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 30, 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.584 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.585 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-16T15:19:03.586 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1920211368 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.587 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/1920211368", 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/1920211368", 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1920211368", 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1920211368", 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "28dad360-6435-4eca-8975-4d482cf456d8", 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-16T15:19:03.588 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 23, 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 31, 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:19:03.589 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.590 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.591 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:19:03.592 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 723941755 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/723941755", 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/723941755", 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/723941755", 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/723941755", 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:19:03.593 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "9e9bbafa-8426-4f88-a475-b4a41f336294", 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 38, 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 0, 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 32, 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.594 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.595 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.596 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:19:03.597 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1944839882 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/1944839882", 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/1944839882", 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1944839882", 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1944839882", 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:19:03.598 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:17:33.503177+0000", 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:15:30.624524+0000", 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.599 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:17:34.101398+0000", 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:16:06.154172+0000", 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:17:34.743868+0000", 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:19:03.600 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:16:42.396512+0000", 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-16T15:19:03.601 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-16T15:19:03.602 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/1920211368,v1:127.0.0.1:6805/1920211368]", 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:19:03.603 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:19:03.604 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:15:03.606999Z", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/1920211368,v1:127.0.0.1:6803/1920211368]", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1920211368,v1:127.0.0.1:6809/1920211368]", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1920211368,v1:127.0.0.1:6807/1920211368]", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:19:03.605 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:19:03.606 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/723941755,v1:127.0.0.1:6813/723941755]", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:19:03.607 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:15:37.864050Z", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:19:03.608 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/723941755,v1:127.0.0.1:6811/723941755]", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/723941755,v1:127.0.0.1:6817/723941755]", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/723941755,v1:127.0.0.1:6815/723941755]", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:19:03.609 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/1944839882,v1:127.0.0.1:6813/1944839882]", 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:19:03.610 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:19:03.611 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:16:13.608965Z", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/1944839882,v1:127.0.0.1:6811/1944839882]", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1944839882,v1:127.0.0.1:6817/1944839882]", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1944839882,v1:127.0.0.1:6815/1944839882]", 2024-09-16T15:19:03.612 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.613 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [] 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 36 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 38 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.614 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 41, 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.615 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.616 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:19:03.616 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-16T15:19:03.616 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:19:03.616 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.616 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.616 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-16T15:19:03.616 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.616 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-16T15:19:03.616 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-16T15:19:03.616 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.616 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.617 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:19:03.617 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-16T15:19:03.617 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.617 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.617 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-16T15:19:03.617 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-16T15:19:03.617 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.617 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.617 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-16T15:19:03.617 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-16T15:19:03.617 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-16T15:19:03.618 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:19:03.619 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:19:03.620 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.621 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:19:03.622 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:19:03.623 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.624 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.625 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-16T15:19:03.626 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-16T15:19:03.627 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 0, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-16T15:19:03.628 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T15:19:03.629 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2" 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-16T15:19:03.630 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 1, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62050, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:19:03.631 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:19:03.632 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:19:03.633 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:19:03.634 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 17, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 17, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:19:03.635 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83624, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2104, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489176, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036916224, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2154496, 2024-09-16T15:19:03.636 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 460431, 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.637 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-16T15:19:03.638 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83624, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2104, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489176, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036916224, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2154496, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 460431, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:19:03.639 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T15:19:03.640 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62050, 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-16T15:19:03.641 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:19:03.642 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:19:03.643 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1220608, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 185266, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:19:03.644 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 17, 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 17, 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 154618822677 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.645 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672993 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 163208757268 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 1 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.646 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:19:03.647 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 7, 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:18:48.326045+0000", 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 112, 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-16T15:19:03.648 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 435, 2024-09-16T15:19:03.649 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-16T15:19:03.649 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:19:03.649 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 593, 2024-09-16T15:19:03.649 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-16T15:19:03.649 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-16T15:19:03.649 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:19:03.649 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:19:03.649 INFO:tasks.workunit.client.0.smithi039.stderr:report 3433486405 2024-09-16T15:19:03.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T15:19:03.650 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery 2024-09-16T15:19:03.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-09-16T15:19:03.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:19:04.002 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:19:04.002 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-16T15:17:48.921357+0000 41'101 41:125 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:08:45.081625+0000 0 0 2024-09-16T15:19:04.002 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:19:04.002 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:19:04.002 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:19:04.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-16T15:19:04.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-09-16T15:19:04.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T15:19:04.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T15:19:04.020 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:19:04.020 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:04.020 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:04.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.0.asok 2024-09-16T15:19:04.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-16T15:19:04.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.0.asok set_recovery_delay 100000 2024-09-16T15:19:04.127 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:19:04.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-16T15:19:04.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-09-16T15:19:04.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T15:19:04.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T15:19:04.141 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:19:04.141 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:04.141 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:04.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.2.asok 2024-09-16T15:19:04.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-16T15:19:04.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.2.asok set_recovery_delay 100000 2024-09-16T15:19:04.259 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:19:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:revive divergent 1 2024-09-16T15:19:04.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-09-16T15:19:04.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:19:04.691 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:19:04.691 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-16T15:17:48.921357+0000 41'101 41:125 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:08:45.081625+0000 0 0 2024-09-16T15:19:04.691 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:19:04.691 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:19:04.691 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:19:04.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-09-16T15:19:05.417 INFO:tasks.workunit.client.0.smithi039.stderr:noup is set 2024-09-16T15:19:05.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-16T15:19:05.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:19:05.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:19:05.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:19:05.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:19:05.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:19:05.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:19:05.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:19:05.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:19:05.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:19:05.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:19:05.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:19:05.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:19:05.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:19:05.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:19:05.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:19:05.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:19:05.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:19:05.443 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:19:05.443 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:05.443 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:05.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:19:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:19:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:19:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:19:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:19:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:19:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:19:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:19:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:19:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:19:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:19:05.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:19:05.445 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:19:05.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:19:05.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:19:05.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-16T15:19:05.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:19:05.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:19:05.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:19:05.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:19:05.487 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:05.484+0000 7f5f15aad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:05.490 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:05.488+0000 7f5f15aad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:05.491 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:05.488+0000 7f5f15aad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:05.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-16T15:19:05.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-09-16T15:19:06.026 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:06.024+0000 7f5f15aad7c0 -1 Falling back to public interface 2024-09-16T15:19:09.108 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:09.108+0000 7f5f15aad7c0 -1 osd.1 29 log_to_monitors true 2024-09-16T15:19:10.892 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery divergent 2024-09-16T15:19:10.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-09-16T15:19:10.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:19:11.300 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:19:11.300 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-16T15:17:48.921357+0000 41'101 42:127 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:08:45.081625+0000 0 0 2024-09-16T15:19:11.300 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:19:11.300 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:19:11.300 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:19:11.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-09-16T15:19:11.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T15:19:11.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T15:19:11.317 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:19:11.317 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:11.317 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:11.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.1.asok 2024-09-16T15:19:11.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-09-16T15:19:11.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.1.asok set_recovery_delay 100000 2024-09-16T15:19:11.424 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:19:11.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-09-16T15:19:12.290 INFO:tasks.workunit.client.0.smithi039.stderr:noup is unset 2024-09-16T15:19:12.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-09-16T15:19:12.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:19:12.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:19:12.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:19:12.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:19:12.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:19:12.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:19:12.312 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:19:12.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:19:12.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:19:12.968 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1920211368,v1:127.0.0.1:6803/1920211368] [v2:127.0.0.1:6804/1920211368,v1:127.0.0.1:6805/1920211368] exists,up fd12a0dc-6b50-4591-8f3b-ac39e6d7839e 2024-09-16T15:19:12.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:19:12.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:19:12.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:19:12.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-09-16T15:19:12.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:19:12.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:19:12.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:19:12.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:19:12.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:19:12.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:19:12.970 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:19:12.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:19:12.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:19:13.415 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/65037186,v1:127.0.0.1:6819/65037186] [v2:127.0.0.1:6820/65037186,v1:127.0.0.1:6821/65037186] exists,up 28dad360-6435-4eca-8975-4d482cf456d8 2024-09-16T15:19:13.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:19:13.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:19:13.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:19:13.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-09-16T15:19:13.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:19:13.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:19:13.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:19:13.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:19:13.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:19:13.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:19:13.417 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:19:13.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:19:13.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:19:13.845 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/1944839882,v1:127.0.0.1:6811/1944839882] [v2:127.0.0.1:6812/1944839882,v1:127.0.0.1:6813/1944839882] exists,up 9e9bbafa-8426-4f88-a475-b4a41f336294 2024-09-16T15:19:13.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:19:13.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:19:13.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:19:13.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:19:14.272 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:19:14.273 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-16T15:17:48.921357+0000 41'101 43:128 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:08:45.081625+0000 0 0 2024-09-16T15:19:14.273 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:19:14.273 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:19:14.273 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:19:14.289 INFO:tasks.workunit.client.0.smithi039.stdout:wait for peering 2024-09-16T15:19:14.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-09-16T15:19:14.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:19:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:19:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 4 4 0 0 62050 0 0 11 90 11 active+recovering+degraded 2024-09-16T15:19:12.962247+0000 41'101 47:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T16:16:39.328631+0000 0 0 2024-09-16T15:19:14.696 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:19:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:19:14.696 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:19:14.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-09-16T15:19:14.816 INFO:tasks.workunit.client.0.smithi039.stdout:killing divergent 1 2024-09-16T15:19:14.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-09-16T15:19:14.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:19:15.226 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:19:15.226 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 4 4 0 0 62050 0 0 11 90 11 active+recovering+degraded 2024-09-16T15:19:12.962247+0000 41'101 47:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T16:16:39.328631+0000 0 0 2024-09-16T15:19:15.226 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:19:15.226 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:19:15.226 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:19:15.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-09-16T15:19:15.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:19:15.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:19:15.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:19:15.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:19:15.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:19:15.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:19:15.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.27113.out 2024-09-16T15:19:15.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27113.out 2024-09-16T15:19:15.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-16T15:19:15.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T15:19:15.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T15:19:15.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T15:19:15.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-16T15:19:15.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27113.out 2024-09-16T15:19:16.635 INFO:tasks.workunit.client.0.smithi039.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-16T15:19:16.644 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:023dd3d2:::existing_12:head# 2024-09-16T15:19:16.673 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:04c40a0c:::existing_93:head# 2024-09-16T15:19:16.700 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:064ff5fb:::existing_36:head# 2024-09-16T15:19:16.718 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:091f9f1d:::existing_94:head# 2024-09-16T15:19:16.726 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0a865a93:::existing_54:head# 2024-09-16T15:19:16.741 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0fed513c:::existing_41:head# 2024-09-16T15:19:16.753 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-09-16T15:19:16.753 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:169157fa:::existing_81:head# 2024-09-16T15:19:16.753 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:17731382:::existing_58:head# 2024-09-16T15:19:16.753 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:18a45a39:::existing_43:head# 2024-09-16T15:19:16.753 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1c160f01:::existing_66:head# 2024-09-16T15:19:16.754 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1dd6c249:::existing_97:head# 2024-09-16T15:19:16.754 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:227e0c77:::existing_7:head# 2024-09-16T15:19:16.755 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:23e1b063:::existing_59:head# 2024-09-16T15:19:16.755 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:28566edb:::existing_78:head# 2024-09-16T15:19:16.756 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2a101182:::existing_20:head# 2024-09-16T15:19:16.756 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2d1519da:::existing_96:head# 2024-09-16T15:19:16.756 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3078b87a:::existing_86:head# 2024-09-16T15:19:16.757 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:30bde01b:::existing_15:head# 2024-09-16T15:19:16.757 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:317a7e19:::existing_56:head# 2024-09-16T15:19:16.758 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-09-16T15:19:16.758 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3e16d84b:::existing_48:head# 2024-09-16T15:19:16.758 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:45fe8203:::existing_75:head# 2024-09-16T15:19:16.759 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-09-16T15:19:16.759 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5298dcdb:::existing_57:head# 2024-09-16T15:19:16.760 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:52ebaa44:::existing_32:head# 2024-09-16T15:19:16.761 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:53588912:::existing_9:head# 2024-09-16T15:19:16.762 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-09-16T15:19:16.762 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5da0255c:::existing_13:head# 2024-09-16T15:19:16.763 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5da2dab8:::existing_53:head# 2024-09-16T15:19:16.763 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5ee3514b:::existing_30:head# 2024-09-16T15:19:16.764 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:602f83fe:::foo:head# 2024-09-16T15:19:16.764 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:60426cc7:::existing_62:head# 2024-09-16T15:19:16.764 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:628918ca:::existing_79:head# 2024-09-16T15:19:16.765 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-09-16T15:19:16.765 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:67e1780f:::existing_27:head# 2024-09-16T15:19:16.766 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:68143ca1:::existing_71:head# 2024-09-16T15:19:16.766 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6bac9509:::existing_29:head# 2024-09-16T15:19:16.767 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6bbec798:::existing_45:head# 2024-09-16T15:19:16.767 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6cb23a40:::existing_65:head# 2024-09-16T15:19:16.767 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7055ec5e:::existing_83:head# 2024-09-16T15:19:16.768 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:70a54e2b:::existing_87:head# 2024-09-16T15:19:16.768 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:750ace21:::existing_19:head# 2024-09-16T15:19:16.769 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:78623b1d:::existing_6:head# 2024-09-16T15:19:16.770 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:79881152:::existing_63:head# 2024-09-16T15:19:16.770 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-09-16T15:19:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:81490677:::existing_76:head# 2024-09-16T15:19:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-09-16T15:19:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-09-16T15:19:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-09-16T15:19:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:923c5d4d:::existing_5:head# 2024-09-16T15:19:16.773 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92731e71:::existing_89:head# 2024-09-16T15:19:16.774 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:928205cf:::existing_82:head# 2024-09-16T15:19:16.774 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:93be2267:::existing_24:head# 2024-09-16T15:19:16.774 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:99ceda2c:::existing_44:head# 2024-09-16T15:19:16.775 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9dd917ee:::existing_26:head# 2024-09-16T15:19:16.775 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a28805a6:::existing_72:head# 2024-09-16T15:19:16.776 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a294a8f6:::existing_16:head# 2024-09-16T15:19:16.776 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a2f5473a:::existing_42:head# 2024-09-16T15:19:16.777 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a504a533:::existing_90:head# 2024-09-16T15:19:16.778 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a691337e:::existing_61:head# 2024-09-16T15:19:16.778 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-09-16T15:19:16.778 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a9fde86e:::existing_18:head# 2024-09-16T15:19:16.779 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:aa309faa:::existing_68:head# 2024-09-16T15:19:16.780 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-09-16T15:19:16.780 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b279b1fe:::existing_10:head# 2024-09-16T15:19:16.780 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b343e277:::existing_88:head# 2024-09-16T15:19:16.781 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b7130e90:::existing_80:head# 2024-09-16T15:19:16.781 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bc6f7309:::existing_70:head# 2024-09-16T15:19:16.781 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:be8489b7:::existing_38:head# 2024-09-16T15:19:16.782 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-09-16T15:19:16.782 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bebec877:::existing_74:head# 2024-09-16T15:19:16.783 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c04347d8:::existing_60:head# 2024-09-16T15:19:16.783 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c156033e:::existing_100:head# 2024-09-16T15:19:16.784 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-09-16T15:19:16.784 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-09-16T15:19:16.785 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-09-16T15:19:16.785 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-09-16T15:19:16.785 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ccb3865b:::existing_73:head# 2024-09-16T15:19:16.786 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cd84e19e:::existing_40:head# 2024-09-16T15:19:16.786 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-09-16T15:19:16.787 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d41a25c9:::existing_22:head# 2024-09-16T15:19:16.788 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d7f088da:::existing_91:head# 2024-09-16T15:19:16.788 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-16T15:19:16.789 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-16T15:19:16.789 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-16T15:19:16.789 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-16T15:19:16.790 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e0e22077:::existing_21:head# 2024-09-16T15:19:16.790 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e255b710:::existing_55:head# 2024-09-16T15:19:16.791 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:eff1d5db:::existing_14:head# 2024-09-16T15:19:16.791 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f1b156bf:::existing_23:head# 2024-09-16T15:19:16.792 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f5ce4199:::existing_34:head# 2024-09-16T15:19:16.792 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f6809456:::existing_11:head# 2024-09-16T15:19:16.793 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f699a2a9:::existing_67:head# 2024-09-16T15:19:16.794 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f9baac43:::existing_25:head# 2024-09-16T15:19:16.794 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fb76688e:::existing_51:head# 2024-09-16T15:19:16.794 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fbb92eec:::existing_69:head# 2024-09-16T15:19:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-16T15:19:16.795 INFO:tasks.workunit.client.0.smithi039.stderr:Export successful 2024-09-16T15:19:16.795 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-16T15:19:16.795 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-16T15:19:17.060 INFO:tasks.workunit.client.0.smithi039.stderr:Remove successful 2024-09-16T15:19:17.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.27113.out 2024-09-16T15:19:17.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-16T15:19:17.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T15:19:17.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T15:19:17.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T15:19:17.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-16T15:19:17.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.27113.out 2024-09-16T15:19:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={23=1}} deleted_pools ) 2024-09-16T15:19:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.0 2024-09-16T15:19:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:023dd3d2:::existing_12:head# 2024-09-16T15:19:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:18.654 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:04c40a0c:::existing_93:head# 2024-09-16T15:19:18.654 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:18.721 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:064ff5fb:::existing_36:head# 2024-09-16T15:19:18.721 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:18.788 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:091f9f1d:::existing_94:head# 2024-09-16T15:19:18.788 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:18.830 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0a865a93:::existing_54:head# 2024-09-16T15:19:18.830 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:18.864 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0fed513c:::existing_41:head# 2024-09-16T15:19:18.864 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:18.897 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-09-16T15:19:18.897 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:18.930 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:169157fa:::existing_81:head# 2024-09-16T15:19:18.930 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:18.964 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:17731382:::existing_58:head# 2024-09-16T15:19:18.964 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:18.997 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:18a45a39:::existing_43:head# 2024-09-16T15:19:18.998 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.031 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1c160f01:::existing_66:head# 2024-09-16T15:19:19.031 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.073 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1dd6c249:::existing_97:head# 2024-09-16T15:19:19.073 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.115 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:227e0c77:::existing_7:head# 2024-09-16T15:19:19.115 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.148 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:23e1b063:::existing_59:head# 2024-09-16T15:19:19.148 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.190 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:28566edb:::existing_78:head# 2024-09-16T15:19:19.190 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.232 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2a101182:::existing_20:head# 2024-09-16T15:19:19.232 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.265 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2d1519da:::existing_96:head# 2024-09-16T15:19:19.265 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.307 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3078b87a:::existing_86:head# 2024-09-16T15:19:19.307 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.340 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:30bde01b:::existing_15:head# 2024-09-16T15:19:19.340 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.382 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:317a7e19:::existing_56:head# 2024-09-16T15:19:19.382 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.415 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-09-16T15:19:19.416 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.474 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3e16d84b:::existing_48:head# 2024-09-16T15:19:19.474 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.516 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:45fe8203:::existing_75:head# 2024-09-16T15:19:19.516 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.549 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-09-16T15:19:19.549 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.583 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5298dcdb:::existing_57:head# 2024-09-16T15:19:19.583 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.616 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:52ebaa44:::existing_32:head# 2024-09-16T15:19:19.616 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.650 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53588912:::existing_9:head# 2024-09-16T15:19:19.650 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.683 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-09-16T15:19:19.683 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.717 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5da0255c:::existing_13:head# 2024-09-16T15:19:19.717 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.750 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5da2dab8:::existing_53:head# 2024-09-16T15:19:19.750 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.783 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ee3514b:::existing_30:head# 2024-09-16T15:19:19.784 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.817 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:602f83fe:::foo:head# 2024-09-16T15:19:19.817 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.850 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:60426cc7:::existing_62:head# 2024-09-16T15:19:19.850 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.884 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:628918ca:::existing_79:head# 2024-09-16T15:19:19.884 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.917 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-09-16T15:19:19.917 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:19.959 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:67e1780f:::existing_27:head# 2024-09-16T15:19:19.959 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.001 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:68143ca1:::existing_71:head# 2024-09-16T15:19:20.001 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.034 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6bac9509:::existing_29:head# 2024-09-16T15:19:20.034 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.084 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6bbec798:::existing_45:head# 2024-09-16T15:19:20.084 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.126 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6cb23a40:::existing_65:head# 2024-09-16T15:19:20.126 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.168 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7055ec5e:::existing_83:head# 2024-09-16T15:19:20.168 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.210 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:70a54e2b:::existing_87:head# 2024-09-16T15:19:20.210 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.252 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:750ace21:::existing_19:head# 2024-09-16T15:19:20.252 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.302 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:78623b1d:::existing_6:head# 2024-09-16T15:19:20.302 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.343 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:79881152:::existing_63:head# 2024-09-16T15:19:20.344 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.405 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-09-16T15:19:20.405 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.477 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81490677:::existing_76:head# 2024-09-16T15:19:20.477 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.519 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-09-16T15:19:20.519 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.561 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-09-16T15:19:20.561 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.603 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-09-16T15:19:20.603 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.644 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:923c5d4d:::existing_5:head# 2024-09-16T15:19:20.644 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.711 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92731e71:::existing_89:head# 2024-09-16T15:19:20.712 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.770 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:928205cf:::existing_82:head# 2024-09-16T15:19:20.770 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.845 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:93be2267:::existing_24:head# 2024-09-16T15:19:20.846 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.887 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:99ceda2c:::existing_44:head# 2024-09-16T15:19:20.888 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.929 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9dd917ee:::existing_26:head# 2024-09-16T15:19:20.929 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:20.971 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a28805a6:::existing_72:head# 2024-09-16T15:19:20.971 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.021 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a294a8f6:::existing_16:head# 2024-09-16T15:19:21.021 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.063 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a2f5473a:::existing_42:head# 2024-09-16T15:19:21.063 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.105 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a504a533:::existing_90:head# 2024-09-16T15:19:21.105 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.146 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a691337e:::existing_61:head# 2024-09-16T15:19:21.147 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.188 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-09-16T15:19:21.188 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.230 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a9fde86e:::existing_18:head# 2024-09-16T15:19:21.230 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.272 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aa309faa:::existing_68:head# 2024-09-16T15:19:21.272 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.313 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-09-16T15:19:21.314 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.389 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b279b1fe:::existing_10:head# 2024-09-16T15:19:21.389 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.430 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b343e277:::existing_88:head# 2024-09-16T15:19:21.431 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.473 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b7130e90:::existing_80:head# 2024-09-16T15:19:21.473 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.506 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bc6f7309:::existing_70:head# 2024-09-16T15:19:21.506 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.539 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:be8489b7:::existing_38:head# 2024-09-16T15:19:21.539 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.573 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-09-16T15:19:21.573 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.606 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bebec877:::existing_74:head# 2024-09-16T15:19:21.606 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.640 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c04347d8:::existing_60:head# 2024-09-16T15:19:21.640 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.673 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c156033e:::existing_100:head# 2024-09-16T15:19:21.673 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.707 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-09-16T15:19:21.707 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.748 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-09-16T15:19:21.749 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.782 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-09-16T15:19:21.782 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.815 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-09-16T15:19:21.815 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.849 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ccb3865b:::existing_73:head# 2024-09-16T15:19:21.849 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.891 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cd84e19e:::existing_40:head# 2024-09-16T15:19:21.891 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.924 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-09-16T15:19:21.924 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.966 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d41a25c9:::existing_22:head# 2024-09-16T15:19:21.966 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:21.999 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d7f088da:::existing_91:head# 2024-09-16T15:19:21.999 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.041 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-16T15:19:22.041 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.083 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-16T15:19:22.083 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.116 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-16T15:19:22.116 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.158 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-16T15:19:22.158 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.208 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e0e22077:::existing_21:head# 2024-09-16T15:19:22.208 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.242 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e255b710:::existing_55:head# 2024-09-16T15:19:22.242 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.275 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:eff1d5db:::existing_14:head# 2024-09-16T15:19:22.275 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.317 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f1b156bf:::existing_23:head# 2024-09-16T15:19:22.317 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.350 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f5ce4199:::existing_34:head# 2024-09-16T15:19:22.350 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.384 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f6809456:::existing_11:head# 2024-09-16T15:19:22.384 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.451 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f699a2a9:::existing_67:head# 2024-09-16T15:19:22.451 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.509 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f9baac43:::existing_25:head# 2024-09-16T15:19:22.510 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.543 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fb76688e:::existing_51:head# 2024-09-16T15:19:22.543 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.576 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fbb92eec:::existing_69:head# 2024-09-16T15:19:22.576 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:19:22.609 INFO:tasks.workunit.client.0.smithi039.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-16T15:19:22.651 INFO:tasks.workunit.client.0.smithi039.stdout:Import successful 2024-09-16T15:19:23.198 INFO:tasks.workunit.client.0.smithi039.stdout:reviving divergent 1 2024-09-16T15:19:23.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-09-16T15:19:23.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:19:23.616 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:19:23.616 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 4 4 0 0 62050 0 0 11 90 11 active+recovering+degraded 2024-09-16T15:19:12.962247+0000 41'101 47:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T16:16:39.328631+0000 0 0 2024-09-16T15:19:23.616 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:19:23.616 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:19:23.616 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:19:23.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-16T15:19:23.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:19:23.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:19:23.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:19:23.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:19:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:19:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:19:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:19:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:19:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:19:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:19:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:19:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:19:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:19:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:19:23.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:19:23.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:19:23.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:19:23.637 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:19:23.637 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:23.637 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:23.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:19:23.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:19:23.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:19:23.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:19:23.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:19:23.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:19:23.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:19:23.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:19:23.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:19:23.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:19:23.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:19:23.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:19:23.639 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:19:23.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:19:23.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:19:23.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-16T15:19:23.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:19:23.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:19:23.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:19:23.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:19:23.678 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:23.676+0000 7f0febec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:23.681 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:23.680+0000 7f0febec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:23.682 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:23.680+0000 7f0febec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:24.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T15:19:24.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:19:24.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:19:24.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:19:24.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:19:24.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:19:24.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:19:24.054 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:19:24.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:19:24.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:19:24.249 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:24.248+0000 7f0febec47c0 -1 Falling back to public interface 2024-09-16T15:19:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/65037186,v1:127.0.0.1:6819/65037186] [v2:127.0.0.1:6820/65037186,v1:127.0.0.1:6821/65037186] exists,up 28dad360-6435-4eca-8975-4d482cf456d8 2024-09-16T15:19:24.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:19:24.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:19:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:19:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-09-16T15:19:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:19:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:19:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:19:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:19:24.479 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:19:24.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:19:24.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:19:24.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:19:24.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:19:24.953 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/65037186,v1:127.0.0.1:6819/65037186] [v2:127.0.0.1:6820/65037186,v1:127.0.0.1:6821/65037186] exists,up 28dad360-6435-4eca-8975-4d482cf456d8 2024-09-16T15:19:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:19:24.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:19:24.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:19:24.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-09-16T15:19:27.248 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:27.248+0000 7f0febec47c0 -1 osd.1 47 log_to_monitors true 2024-09-16T15:19:44.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-09-16T15:19:44.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T15:19:44.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T15:19:44.957 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:19:44.957 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:44.957 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:44.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.1.asok 2024-09-16T15:19:44.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-09-16T15:19:44.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T15:19:45.077 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:19:45.077 INFO:tasks.workunit.client.0.smithi039.stdout: "ops": [], 2024-09-16T15:19:45.077 INFO:tasks.workunit.client.0.smithi039.stdout: "num_ops": 0 2024-09-16T15:19:45.077 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:19:45.091 INFO:tasks.workunit.client.0.smithi039.stdout:allowing recovery 2024-09-16T15:19:45.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-09-16T15:19:45.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:19:45.499 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:19:45.499 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-09-16T15:19:29.150919+0000 47'102 51:150 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:10:51.218648+0000 0 0 2024-09-16T15:19:45.499 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:19:45.500 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:19:45.500 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:19:45.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-16T15:19:45.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-16T15:19:45.703 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:19:45.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-16T15:19:45.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-16T15:19:45.872 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:19:45.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-16T15:19:45.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-16T15:19:46.038 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:19:46.055 INFO:tasks.workunit.client.0.smithi039.stdout:reading divergent objects 2024-09-16T15:19:46.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-09-16T15:19:46.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-09-16T15:19:46.442 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:19:46.442 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-09-16T15:19:29.150919+0000 47'102 51:150 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:16:45.994026+0000 0'0 2024-09-16T15:16:45.994026+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:10:51.218648+0000 0 0 2024-09-16T15:19:46.442 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:19:46.442 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:19:46.442 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:19:46.459 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-09-16T15:19:46.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-09-16T15:19:46.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:19:46.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-09-16T15:19:46.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:19:46.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-09-16T15:19:46.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:19:46.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-09-16T15:19:46.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:19:46.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-09-16T15:19:46.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-09-16T15:19:46.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-16T15:19:46.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-16T15:19:46.630 INFO:tasks.workunit.client.0.smithi039.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-16T15:19:46.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-16T15:19:46.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-16T15:19:46.683 INFO:tasks.workunit.client.0.smithi039.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-16T15:19:46.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-09-16T15:19:46.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-16T15:19:46.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-16T15:19:46.690 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4631.0:1 2024-09-16T15:17:22.595158+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:19:46.690 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: keeping 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4631.0:1 2024-09-16T15:17:22.595158+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:19:46.690 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:19:46.690 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-09-16T15:19:46.690 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: more recent entry found: 41'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4748.0:1 2024-09-16T15:18:27.012339+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0, already merged 2024-09-16T15:19:46.691 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4616.0:1 2024-09-16T15:17:22.563186+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-16T15:19:46.691 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4616.0:1 2024-09-16T15:17:22.563186+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-16T15:19:46.691 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:19:46.691 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-16T15:19:46.691 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-16T15:19:46.691 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-16T15:19:46.691 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4628.0:1 2024-09-16T15:17:22.592355+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:19:46.691 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4628.0:1 2024-09-16T15:17:22.592355+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:19:46.691 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-16T15:19:46.691 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-09-16T15:19:46.691 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4628.0:1 2024-09-16T15:17:22.592355+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4619.0:1 2024-09-16T15:17:22.573046+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4619.0:1 2024-09-16T15:17:22.573046+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-16T15:19:46.692 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-09-16T15:19:46.693 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4619.0:1 2024-09-16T15:17:22.573046+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:19:46.693 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-16T15:19:46.693 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-09-16T15:19:46.693 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4630.0:1 2024-09-16T15:17:22.592114+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:19:46.693 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: keeping 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4630.0:1 2024-09-16T15:17:22.592114+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:19:46.693 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:19:46.693 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-16T15:19:46.693 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-16T15:19:46.693 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-16T15:19:46.694 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-09-16T15:19:46.694 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4630.0:1 2024-09-16T15:17:22.592114+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:19:46.694 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-16T15:19:46.694 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-09-16T15:19:46.694 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4624.0:1 2024-09-16T15:17:22.588383+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-16T15:19:46.694 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4624.0:1 2024-09-16T15:17:22.588383+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-16T15:19:46.694 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:19:46.694 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-16T15:19:46.694 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-16T15:19:46.694 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:19:12.516+0000 7f5ef9a38640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-16T15:19:46.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-16T15:19:46.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-16T15:19:46.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-16T15:19:46.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-16T15:19:46.702 INFO:tasks.workunit.client.0.smithi039.stdout:success 2024-09-16T15:19:46.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-09-16T15:19:46.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.27113.out 2024-09-16T15:19:46.703 INFO:tasks.workunit.client.0.smithi039.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27113.out': No such file or directory 2024-09-16T15:19:46.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-09-16T15:19:46.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T15:19:46.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T15:19:47.317 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T15:19:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-09-16T15:19:47.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:19:47.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:19:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:19:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:19:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:19:47.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:19:47.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-16T15:19:47.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-16T15:19:47.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:19:47.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-16T15:19:47.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:19:47.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:19:47.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:19:47.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:19:47.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:19:47.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:19:47.657 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:19:47.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:19:47.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:19:47.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:19:47.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:19:47.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:19:47.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:19:47.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:19:47.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:19:47.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:19:47.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:19:47.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:19:47.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-16T15:19:47.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:19:47.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:47.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:47.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27113 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:19:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:19:47.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:19:47.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:19:47.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:19:47.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:19:47.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:19:47.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:19:47.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:19:47.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:19:47.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:19:47.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:19:47.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:19:47.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:19:47.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:19:47.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:19:47.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-16T15:19:47.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:19:47.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:47.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:47.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27113 2024-09-16T15:19:47.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:19:47.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:19:47.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-16T15:19:47.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:19:47.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:47.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:47.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27113 2024-09-16T15:19:47.698 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:19:47.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T15:19:47.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:19:47.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-16T15:19:47.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-09-16T15:19:47.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-09-16T15:19:47.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-09-16T15:19:47.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-09-16T15:19:47.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-09-16T15:19:47.699 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-09-16T15:19:47.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-09-16T15:19:47.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-09-16T15:19:47.700 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:19:47.700 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:19:47.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-09-16T15:19:47.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-16T15:19:47.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:19:47.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:19:47.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:19:47.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-16T15:19:47.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-16T15:19:48.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:19:48.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:19:48.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:19:48.067 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:19:48.067 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:48.068 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:48.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:19:48.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:19:48.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:19:48.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:19:48.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:19:48.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:19:48.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:19:48.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:19:48.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:19:48.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:19:48.286 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:19:48.287 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:19:48.287 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:48.287 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:48.287 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-mon.a.asok 2024-09-16T15:19:48.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:19:48.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27113/ceph-mon.a.asok config get fsid 2024-09-16T15:19:48.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:19:48.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:19:48.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:19:48.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:19:48.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:19:48.407 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:19:48.407 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:19:48.407 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:19:48.408 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:19:48.408 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:48.408 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:48.408 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-mon.a.asok 2024-09-16T15:19:48.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:19:48.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27113/ceph-mon.a.asok config get mon_host 2024-09-16T15:19:48.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-09-16T15:19:48.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-16T15:19:48.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:19:48.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:19:48.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:19:48.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-16T15:19:48.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:19:49.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:19:49.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:19:49.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:19:49.456 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:19:49.456 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:49.456 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:49.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:19:49.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:19:49.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:19:49.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-16T15:19:49.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-09-16T15:19:49.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:19:49.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:19:49.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:19:49.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:19:49.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-16T15:19:49.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:19:49.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:19:49.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:19:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:19:49.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-16T15:19:49.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:19:49.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=39492c66-933f-48d1-bb2d-ba158445ce13 2024-09-16T15:19:49.523 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 39492c66-933f-48d1-bb2d-ba158445ce13 2024-09-16T15:19:49.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 39492c66-933f-48d1-bb2d-ba158445ce13' 2024-09-16T15:19:49.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:19:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVTOhm7sq1IBAAvRRUIjjnUzZN/kTs8gyL7g== 2024-09-16T15:19:49.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVTOhm7sq1IBAAvRRUIjjnUzZN/kTs8gyL7g=="}' 2024-09-16T15:19:49.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 39492c66-933f-48d1-bb2d-ba158445ce13 -i td/divergent-priors/0/new.json 2024-09-16T15:19:49.898 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:19:49.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-16T15:19:49.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCVTOhm7sq1IBAAvRRUIjjnUzZN/kTs8gyL7g== --osd-uuid 39492c66-933f-48d1-bb2d-ba158445ce13 2024-09-16T15:19:49.950 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:49.947+0000 7f7e5b2217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:49.984 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:49.983+0000 7f7e5b2217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:49.986 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:49.983+0000 7f7e5b2217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:49.986 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:49.983+0000 7f7e5b2217c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:19:49.986 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:49.983+0000 7f7e5b2217c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:19:49.987 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:49.983+0000 7f7e5b2217c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:19:50.024 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:50.023+0000 7f7e5b2217c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-16T15:19:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-16T15:19:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:19:56.317 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:19:56.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:19:56.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:19:56.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:19:56.937 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:19:56.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:19:56.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:19:56.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:19:56.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:19:56.979 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:56.979+0000 7f51ca5567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:56.989 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:56.987+0000 7f51ca5567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:56.991 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:56.987+0000 7f51ca5567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:19:57.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:19:57.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:19:57.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:19:57.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:19:57.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:19:57.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:19:57.348 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:19:57.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:19:57.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:19:57.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:19:57.565 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:19:57.563+0000 7f51ca5567c0 -1 Falling back to public interface 2024-09-16T15:19:57.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:19:58.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:19:58.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:19:58.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:19:58.781 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:19:58.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:19:58.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:19:59.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:00.098 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:00.095+0000 7f51ca5567c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:20:00.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:00.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:00.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:20:00.216 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:20:00.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:00.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:00.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:01.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:01.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:01.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:20:01.681 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:20:01.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:01.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:02.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:03.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:03.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:03.154 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:20:03.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:20:03.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:03.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:03.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:04.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:04.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:04.603 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:20:04.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:20:04.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:04.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:05.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:06.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:06.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:06.059 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:20:06.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:20:06.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:06.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:06.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:07.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:07.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:07.505 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:20:07.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:20:07.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:07.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:07.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:08.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:08.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:08.963 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:20:08.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:20:08.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:08.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:09.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:10.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:10.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:10.424 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:20:10.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:20:10.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:10.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:10.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:11.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:11.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:11.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:20:11.939 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:20:11.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:11.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:12.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:13.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:13.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:13.382 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:20:13.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:20:13.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:13.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:14.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:15.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:15.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:15.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:20:15.149 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:20:15.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:15.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:16.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:17.151 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:20:17.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:20:17.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:17.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:17.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:18.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:18.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:18.924 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:20:18.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:20:18.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:18.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:20.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:21.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:21.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:21.050 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:20:21.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:20:21.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:21.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:22.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:23.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:23.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:23.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:20:23.186 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:20:23.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:23.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:23.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:24.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:24.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:24.658 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:20:24.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:20:24.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:24.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:20:25.135 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2075863896,v1:127.0.0.1:6803/2075863896] [v2:127.0.0.1:6804/2075863896,v1:127.0.0.1:6805/2075863896] exists,up 39492c66-933f-48d1-bb2d-ba158445ce13 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:20:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:20:25.138 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:20:25.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:20:25.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:20:25.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:20:25.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:20:25.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:20:25.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:20:25.141 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 0c65ed33-581c-493f-b80f-f44168583347 2024-09-16T15:20:25.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c65ed33-581c-493f-b80f-f44168583347 2024-09-16T15:20:25.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0c65ed33-581c-493f-b80f-f44168583347' 2024-09-16T15:20:25.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:20:25.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5TOhmDUtBChAAgq7+ol1q3SOjoRz8lcH8Vw== 2024-09-16T15:20:25.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5TOhmDUtBChAAgq7+ol1q3SOjoRz8lcH8Vw=="}' 2024-09-16T15:20:25.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c65ed33-581c-493f-b80f-f44168583347 -i td/divergent-priors/1/new.json 2024-09-16T15:20:25.721 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:20:25.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-16T15:20:25.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5TOhmDUtBChAAgq7+ol1q3SOjoRz8lcH8Vw== --osd-uuid 0c65ed33-581c-493f-b80f-f44168583347 2024-09-16T15:20:25.785 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:25.783+0000 7ff050db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:20:25.801 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:25.803+0000 7ff050db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:20:25.803 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:25.803+0000 7ff050db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:20:25.803 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:25.803+0000 7ff050db27c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:20:25.803 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:25.803+0000 7ff050db27c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:20:25.804 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:25.803+0000 7ff050db27c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:20:25.885 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:25.887+0000 7ff050db27c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-16T15:20:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-16T15:20:32.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:20:32.083 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:20:32.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:20:32.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:20:32.659 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:20:32.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:20:32.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:20:32.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:20:32.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:20:32.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:20:32.700 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:32.703+0000 7f9da921c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:20:32.702 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:32.703+0000 7f9da921c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:20:32.703 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:32.707+0000 7f9da921c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:20:33.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:20:33.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:20:33.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:20:33.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:20:33.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:20:33.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:33.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:20:33.077 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:20:33.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:33.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:33.249 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:33.251+0000 7f9da921c7c0 -1 Falling back to public interface 2024-09-16T15:20:33.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:34.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:34.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:34.493 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:20:34.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:20:34.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:34.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:34.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:35.816 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:20:35.819+0000 7f9da921c7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:20:35.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:35.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:35.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:20:35.925 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:20:35.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:35.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:36.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:37.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:37.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:37.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:20:37.616 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:20:37.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:37.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:38.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:39.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:39.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:39.053 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:20:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:20:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:39.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:40.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:40.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:40.609 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:20:40.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:20:40.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:40.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:41.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:42.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:42.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:42.137 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:20:42.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:20:42.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:42.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:42.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:43.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:43.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:43.724 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:20:43.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:20:43.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:43.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:44.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:45.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:45.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:45.173 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:20:45.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:20:45.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:45.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:45.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:46.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:46.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:46.681 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:20:46.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:20:46.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:46.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:47.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:48.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:48.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:48.169 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:20:48.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:20:48.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:48.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:48.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:49.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:49.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:49.827 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:20:49.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:20:49.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:49.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:50.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:51.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:51.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:51.554 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:20:51.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:20:51.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:51.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:52.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:53.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:53.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:53.309 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:20:53.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:20:53.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:53.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:54.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:55.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:55.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:55.132 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:20:55.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:20:55.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:55.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:56.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:57.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:57.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:57.096 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:20:57.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:20:57.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:57.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:57.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:20:58.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:20:58.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:20:58.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:20:58.963 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:20:58.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:20:58.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:20:59.459 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3128683758,v1:127.0.0.1:6811/3128683758] [v2:127.0.0.1:6812/3128683758,v1:127.0.0.1:6813/3128683758] exists,up 0c65ed33-581c-493f-b80f-f44168583347 2024-09-16T15:20:59.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:20:59.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:20:59.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:20:59.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-16T15:20:59.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-09-16T15:20:59.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:20:59.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:20:59.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:20:59.463 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:20:59.463 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:20:59.463 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:20:59.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:20:59.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:20:59.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:20:59.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:20:59.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:20:59.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:20:59.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:20:59.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:20:59.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:20:59.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:20:59.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:20:59.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:20:59.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:20:59.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-16T15:20:59.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:20:59.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9c568d2-49f6-4872-a744-3ce12eff3568 2024-09-16T15:20:59.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d9c568d2-49f6-4872-a744-3ce12eff3568' 2024-09-16T15:20:59.465 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 d9c568d2-49f6-4872-a744-3ce12eff3568 2024-09-16T15:20:59.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:20:59.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbTOhmQvFBHRAAMSsxc31r9kjwelz2OGCJ4w== 2024-09-16T15:20:59.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbTOhmQvFBHRAAMSsxc31r9kjwelz2OGCJ4w=="}' 2024-09-16T15:20:59.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9c568d2-49f6-4872-a744-3ce12eff3568 -i td/divergent-priors/2/new.json 2024-09-16T15:21:00.145 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:21:00.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-16T15:21:00.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbTOhmQvFBHRAAMSsxc31r9kjwelz2OGCJ4w== --osd-uuid d9c568d2-49f6-4872-a744-3ce12eff3568 2024-09-16T15:21:00.216 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:00.218+0000 7f728dd287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:21:00.218 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:00.218+0000 7f728dd287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:21:00.219 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:00.222+0000 7f728dd287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:21:00.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:00.222+0000 7f728dd287c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:21:00.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:00.222+0000 7f728dd287c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:21:00.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:00.222+0000 7f728dd287c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:21:00.286 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:00.286+0000 7f728dd287c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-16T15:21:06.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-16T15:21:06.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:21:06.558 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T15:21:06.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T15:21:06.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:21:07.125 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:21:07.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T15:21:07.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:21:07.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:21:07.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:21:07.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:21:07.167 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:07.166+0000 7f77f3b027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:21:07.176 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:07.178+0000 7f77f3b027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:21:07.178 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:07.178+0000 7f77f3b027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:21:07.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T15:21:07.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:21:07.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:21:07.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:21:07.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:21:07.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:07.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:21:07.602 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:21:07.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:07.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:07.753 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:07.754+0000 7f77f3b027c0 -1 Falling back to public interface 2024-09-16T15:21:08.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:09.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:09.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:09.032 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:21:09.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:21:09.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:09.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:09.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:10.338 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:21:10.338+0000 7f77f3b027c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:21:10.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:10.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:10.466 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:21:10.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:21:10.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:10.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:10.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:11.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:11.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:11.933 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:21:11.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:21:11.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:11.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:12.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:13.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:13.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:13.422 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:21:13.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:21:13.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:13.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:13.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:14.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:14.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:14.856 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:21:14.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:21:14.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:14.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:15.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:16.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:16.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:16.405 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:21:16.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:21:16.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:16.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:16.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:17.961 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:21:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:21:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:17.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:18.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:19.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:19.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:19.579 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:21:19.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:21:19.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:19.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:20.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:21.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:21.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:21.073 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:21:21.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:21:21.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:21.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:21.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:22.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:22.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:22.650 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:21:22.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:21:22.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:22.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:23.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:24.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:24.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:24.266 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:21:24.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:21:24.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:24.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:24.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:25.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:25.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:25.751 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:21:25.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:21:25.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:25.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:26.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:27.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:27.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:27.480 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:21:27.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:21:27.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:27.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:28.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:29.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:29.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:29.183 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:21:29.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:21:29.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:29.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:29.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:30.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:30.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:30.886 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:21:30.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:21:30.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:30.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:31.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:32.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:32.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:32.507 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:21:32.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:21:32.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:32.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:33.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:21:34.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:21:34.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:21:34.277 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:21:34.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:21:34.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:21:34.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:21:34.710 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1109272698,v1:127.0.0.1:6819/1109272698] [v2:127.0.0.1:6820/1109272698,v1:127.0.0.1:6821/1109272698] exists,up d9c568d2-49f6-4872-a744-3ce12eff3568 2024-09-16T15:21:34.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:21:34.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:21:34.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:21:34.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-09-16T15:21:35.436 INFO:tasks.workunit.client.0.smithi039.stderr:noout is set 2024-09-16T15:21:35.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-09-16T15:21:36.391 INFO:tasks.workunit.client.0.smithi039.stderr:noin is set 2024-09-16T15:21:36.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-09-16T15:21:37.379 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T15:21:37.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-09-16T15:21:37.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-16T15:21:37.949 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T15:21:37.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:21:38.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-09-16T15:21:39.815 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 3 2024-09-16T15:21:39.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-09-16T15:21:40.821 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 2 2024-09-16T15:21:40.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-09-16T15:21:41.768 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 pg_autoscale_mode to on 2024-09-16T15:21:41.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-09-16T15:21:41.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-09-16T15:21:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1726500101 2024-09-16T15:21:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-09-16T15:21:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-09-16T15:21:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-09-16T15:21:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:21:41.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:21:42.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:21:42.241 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:21:42.241 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:21:42.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:21:42.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:21:42.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:21:42.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836504 2024-09-16T15:21:42.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836504 2024-09-16T15:21:42.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504' 2024-09-16T15:21:42.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:21:42.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:21:42.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672975 2024-09-16T15:21:42.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672975 2024-09-16T15:21:42.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504 1-42949672975' 2024-09-16T15:21:42.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:21:42.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:21:42.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509448 2024-09-16T15:21:42.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509448 2024-09-16T15:21:42.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504 1-42949672975 2-64424509448' 2024-09-16T15:21:42.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:21:42.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836504 2024-09-16T15:21:42.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:21:42.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:21:42.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836504 2024-09-16T15:21:42.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:21:42.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836504 2024-09-16T15:21:42.789 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836504 2024-09-16T15:21:42.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-16T15:21:42.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:21:43.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-16T15:21:43.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:21:44.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:21:44.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:21:44.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-16T15:21:44.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:21:44.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672975 2024-09-16T15:21:44.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:21:44.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:21:44.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672975 2024-09-16T15:21:44.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:21:44.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672975 2024-09-16T15:21:44.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-16T15:21:44.654 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672975 2024-09-16T15:21:44.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:21:45.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-16T15:21:45.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:21:45.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509448 2024-09-16T15:21:45.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:21:45.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:21:45.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509448 2024-09-16T15:21:45.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:21:45.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509448 2024-09-16T15:21:45.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-16T15:21:45.065 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509448 2024-09-16T15:21:45.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:21:45.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-16T15:21:45.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-09-16T15:21:45.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:21:45.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:21:45.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:21:45.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:21:45.492 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:21:45.492 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:21:45.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:21:45.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:21:45.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:21:45.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:21:45.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:21:45.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:21:45.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:21:45.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:21:45.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:21:46.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:21:46.027 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:21:46.027 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:21:46.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:21:46.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:21:46.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:21:46.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836507 2024-09-16T15:21:46.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836507 2024-09-16T15:21:46.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836507' 2024-09-16T15:21:46.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:21:46.195 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:21:46.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672978 2024-09-16T15:21:46.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672978 2024-09-16T15:21:46.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836507 1-42949672978' 2024-09-16T15:21:46.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:21:46.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:21:46.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509451 2024-09-16T15:21:46.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509451 2024-09-16T15:21:46.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836507 1-42949672978 2-64424509451' 2024-09-16T15:21:46.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:21:46.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836507 2024-09-16T15:21:46.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:21:46.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:21:46.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836507 2024-09-16T15:21:46.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:21:46.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836507 2024-09-16T15:21:46.520 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836507 2024-09-16T15:21:46.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-16T15:21:46.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:21:46.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-16T15:21:46.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:21:46.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672978 2024-09-16T15:21:46.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:21:46.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:21:46.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672978 2024-09-16T15:21:46.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:21:46.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672978 2024-09-16T15:21:46.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-16T15:21:46.981 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672978 2024-09-16T15:21:46.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:21:47.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672976 -lt 42949672978 2024-09-16T15:21:47.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:21:48.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:21:48.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:21:48.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-16T15:21:48.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:21:48.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509451 2024-09-16T15:21:48.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:21:48.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:21:48.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509451 2024-09-16T15:21:48.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:21:48.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509451 2024-09-16T15:21:48.863 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509451 2024-09-16T15:21:48.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-16T15:21:48.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:21:49.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-16T15:21:49.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:21:49.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:21:49.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:21:49.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:21:49.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:21:49.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:21:49.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:21:49.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:21:49.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:21:49.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:21:49.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:21:50.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T15:21:50.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:21:50.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:21:50.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:21:50.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T15:21:50.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:21:50.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:21:50.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-09-16T15:21:50.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-09-16T15:21:51.218 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:21:51.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-09-16T15:21:51.234 INFO:tasks.workunit.client.0.smithi039.stdout:primary and soon to be divergent is 1 2024-09-16T15:21:51.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-09-16T15:21:51.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:21:51.627 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:21:51.627 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T15:21:39.658106+0000 0'0 29:16 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:36:37.175570+0000 0 0 2024-09-16T15:21:51.627 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:21:51.627 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:21:51.627 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:21:51.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-09-16T15:21:51.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1726500111 2024-09-16T15:21:51.644 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1726500111 - 1726500101 2024-09-16T15:21:51.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=10 2024-09-16T15:21:51.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 10 -gt 300 ']' 2024-09-16T15:21:51.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-09-16T15:21:51.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-09-16T15:21:51.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-16T15:21:51.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-09-16T15:21:51.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-09-16T15:21:51.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-16T15:21:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-09-16T15:21:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-09-16T15:21:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-16T15:21:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-09-16T15:21:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-09-16T15:21:51.647 INFO:tasks.workunit.client.0.smithi039.stdout:writing initial objects 2024-09-16T15:21:51.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-09-16T15:21:51.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-09-16T15:21:51.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:51.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-09-16T15:21:51.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:51.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-09-16T15:21:52.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:52.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-09-16T15:21:52.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:52.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-09-16T15:21:52.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:52.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-09-16T15:21:52.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:52.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-09-16T15:21:52.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:52.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-09-16T15:21:52.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:52.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-09-16T15:21:53.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:53.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-09-16T15:21:53.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:53.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-09-16T15:21:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:53.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-09-16T15:21:53.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:53.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-09-16T15:21:53.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:53.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-09-16T15:21:53.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:53.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-09-16T15:21:54.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:54.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-09-16T15:21:54.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:54.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-09-16T15:21:54.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:54.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-09-16T15:21:54.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:54.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-09-16T15:21:54.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:54.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-09-16T15:21:55.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:55.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-09-16T15:21:55.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:55.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-09-16T15:21:55.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:55.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-09-16T15:21:55.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:55.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-09-16T15:21:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-09-16T15:21:55.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:55.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-09-16T15:21:56.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:56.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-09-16T15:21:56.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:56.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-09-16T15:21:56.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:56.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-09-16T15:21:57.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:57.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-09-16T15:21:57.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:57.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-09-16T15:21:58.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:58.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-09-16T15:21:58.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:58.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-09-16T15:21:58.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:58.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-09-16T15:21:58.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:58.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-09-16T15:21:58.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:58.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-09-16T15:21:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-09-16T15:21:59.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:59.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-09-16T15:21:59.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:59.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-09-16T15:21:59.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:59.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-09-16T15:21:59.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:59.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-09-16T15:21:59.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:21:59.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-09-16T15:22:00.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:00.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-09-16T15:22:00.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:00.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-09-16T15:22:00.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:00.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-09-16T15:22:00.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:00.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-09-16T15:22:00.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:00.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-09-16T15:22:01.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:01.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-09-16T15:22:01.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:01.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-09-16T15:22:01.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:01.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-09-16T15:22:01.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:01.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-09-16T15:22:01.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:01.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-09-16T15:22:01.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:01.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-09-16T15:22:02.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:02.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-09-16T15:22:02.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:02.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-09-16T15:22:02.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:02.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-09-16T15:22:03.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:03.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-09-16T15:22:04.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:04.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-09-16T15:22:04.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:04.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-09-16T15:22:04.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:04.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-09-16T15:22:05.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:05.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-09-16T15:22:05.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:05.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-09-16T15:22:05.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:05.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-09-16T15:22:06.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:06.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-09-16T15:22:06.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:06.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-09-16T15:22:06.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:06.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-09-16T15:22:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-09-16T15:22:07.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:07.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-09-16T15:22:08.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:08.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-09-16T15:22:08.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:08.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-09-16T15:22:08.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:08.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-09-16T15:22:08.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:08.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-09-16T15:22:09.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:09.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-09-16T15:22:09.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:09.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-09-16T15:22:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:09.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-09-16T15:22:09.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:09.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-09-16T15:22:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:09.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-09-16T15:22:09.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:09.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-09-16T15:22:10.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:10.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-09-16T15:22:10.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:10.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-09-16T15:22:10.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:10.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-09-16T15:22:10.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:10.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-09-16T15:22:11.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:11.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-09-16T15:22:11.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:11.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-09-16T15:22:11.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:11.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-09-16T15:22:11.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:11.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-09-16T15:22:11.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:11.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-09-16T15:22:11.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:11.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-09-16T15:22:12.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:12.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-09-16T15:22:12.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:12.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-09-16T15:22:12.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:12.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-09-16T15:22:13.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:13.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-09-16T15:22:13.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:13.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-09-16T15:22:14.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:14.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-09-16T15:22:14.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:14.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-09-16T15:22:14.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:14.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-09-16T15:22:14.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:14.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-09-16T15:22:14.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:14.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-09-16T15:22:15.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:15.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-09-16T15:22:15.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:15.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-09-16T15:22:15.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-16T15:22:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-09-16T15:22:15.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T15:22:15.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-09-16T15:22:15.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:22:15.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:22:15.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:22:15.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T15:22:15.670 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:22:15.670 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:22:15.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:22:15.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:22:15.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:22:15.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T15:22:15.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:22:15.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:22:15.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:22:15.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:22:15.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:22:16.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:22:16.253 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:22:16.253 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:22:16.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:22:16.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:22:16.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:22:16.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836515 2024-09-16T15:22:16.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836515 2024-09-16T15:22:16.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836515' 2024-09-16T15:22:16.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:22:16.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:22:16.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672986 2024-09-16T15:22:16.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672986 2024-09-16T15:22:16.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836515 1-42949672986' 2024-09-16T15:22:16.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:22:16.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:22:16.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509459 2024-09-16T15:22:16.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509459 2024-09-16T15:22:16.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836515 1-42949672986 2-64424509459' 2024-09-16T15:22:16.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:22:16.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836515 2024-09-16T15:22:16.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:22:16.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:22:16.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836515 2024-09-16T15:22:16.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:22:16.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836515 2024-09-16T15:22:16.990 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836515 2024-09-16T15:22:16.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-09-16T15:22:16.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:22:17.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836513 -lt 21474836515 2024-09-16T15:22:17.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:22:18.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:22:18.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:22:18.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836515 -lt 21474836515 2024-09-16T15:22:18.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:22:18.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672986 2024-09-16T15:22:18.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:22:18.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:22:18.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672986 2024-09-16T15:22:18.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:22:18.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672986 2024-09-16T15:22:18.920 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672986 2024-09-16T15:22:18.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-09-16T15:22:18.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:22:19.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672986 -lt 42949672986 2024-09-16T15:22:19.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:22:19.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509459 2024-09-16T15:22:19.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:22:19.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:22:19.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509459 2024-09-16T15:22:19.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:22:19.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509459 2024-09-16T15:22:19.370 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509459 2024-09-16T15:22:19.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-16T15:22:19.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:22:20.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-16T15:22:20.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:22:20.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:22:20.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:22:20.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-16T15:22:20.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:22:20.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:22:20.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:22:20.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:22:20.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:22:20.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:22:20.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:22:21.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-16T15:22:21.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:22:21.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:22:21.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:22:21.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-16T15:22:21.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:22:21.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:22:21.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-09-16T15:22:21.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-16T15:22:21.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-16T15:22:21.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-16T15:22:21.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T15:22:22.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-09-16T15:22:22.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-09-16T15:22:22.181 INFO:tasks.workunit.client.0.smithi039.stdout:blackholing osds 0 2 2024-09-16T15:22:22.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:22:22.594 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:22:22.594 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:22:08.049092+0000 39'29 47:77 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.595 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:22:13.724633+0000 44'29 47:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.595 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:22:03.742375+0000 40'31 47:83 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.595 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:22:08.048561+0000 43'29 47:77 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.595 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:22:10.125691+0000 35'29 46:92 [0,1,2] 0 [0,1,2] 0 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:07:07.199754+0000 3 0 2024-09-16T15:22:22.595 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:22:19.722966+0000 41'29 48:71 [1,2,0] 1 [1,2,0] 1 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.595 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-16T15:22:17.394800+0000 43'30 47:95 [1,2,0] 1 [1,2,0] 1 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T03:00:52.797756+0000 3 0 2024-09-16T15:22:22.595 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1138 0 0 16 12 16 active+clean 2024-09-16T15:22:07.997190+0000 35'28 47:71 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.596 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:22:13.724213+0000 40'30 47:74 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.596 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-16T15:22:18.581628+0000 42'32 47:95 [2,0,1] 2 [2,0,1] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:50:02.365463+0000 6 0 2024-09-16T15:22:22.596 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-16T15:22:02.244472+0000 36'28 47:84 [1,0,2] 1 [1,0,2] 1 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:11:18.460354+0000 1 0 2024-09-16T15:22:22.596 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:22:08.048454+0000 44'30 47:88 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.596 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:22:19.240869+0000 40'28 47:77 [0,1,2] 0 [0,1,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T18:28:16.449204+0000 1 0 2024-09-16T15:22:22.596 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:22:12.130441+0000 41'29 47:85 [1,0,2] 1 [1,0,2] 1 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:37:35.561921+0000 3 0 2024-09-16T15:22:22.596 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-16T15:22:04.583920+0000 41'29 47:81 [2,0,1] 2 [2,0,1] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T20:21:45.960892+0000 0 0 2024-09-16T15:22:22.596 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:21:57.577280+0000 33'29 47:73 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.597 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:22:00.096839+0000 43'29 47:85 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:50:42.273513+0000 2 0 2024-09-16T15:22:22.597 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:22:19.648233+0000 41'29 48:81 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:37:46.498548+0000 3 0 2024-09-16T15:22:22.597 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:22:13.724893+0000 41'29 47:74 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.597 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:22:07.996854+0000 39'29 47:77 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.597 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-16T15:22:20.039632+0000 43'30 48:86 [1,0,2] 1 [1,0,2] 1 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T21:07:25.107191+0000 3 0 2024-09-16T15:22:22.597 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:22:07.611353+0000 33'29 47:93 [2,0,1] 2 [2,0,1] 2 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T21:10:08.649524+0000 3 0 2024-09-16T15:22:22.597 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:22:16.229919+0000 33'28 46:89 [0,1,2] 0 [0,1,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:10:01.681706+0000 1 0 2024-09-16T15:22:22.597 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:22:08.048170+0000 43'30 47:78 [1,2,0] 1 [1,2,0] 1 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.598 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:21:57.574840+0000 44'29 47:73 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.598 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:22:12.569098+0000 39'29 47:92 [2,0,1] 2 [2,0,1] 2 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:41:15.732286+0000 3 0 2024-09-16T15:22:22.598 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:22:13.724550+0000 40'31 47:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.598 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 520 0 0 15 12 15 active+clean 2024-09-16T15:22:08.049022+0000 31'27 47:75 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.598 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:22:17.394875+0000 44'30 47:83 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.598 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1137 0 0 16 12 16 active+clean 2024-09-16T15:22:19.722439+0000 33'28 48:67 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.598 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:22:05.800633+0000 41'30 46:79 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.598 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:22:08.047815+0000 44'31 47:90 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:22.598 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:22:22.599 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:22:22.599 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:22:22.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-16T15:22:22.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-09-16T15:22:22.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T15:22:22.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T15:22:22.613 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:22:22.614 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:22:22.614 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:22:22.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.0.asok 2024-09-16T15:22:22.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-16T15:22:22.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-16T15:22:22.738 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:22:22.738 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-16T15:22:22.738 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:22:22.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-16T15:22:22.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-09-16T15:22:22.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T15:22:22.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T15:22:22.753 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:22:22.753 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:22:22.753 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:22:22.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.2.asok 2024-09-16T15:22:22.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-16T15:22:22.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-16T15:22:22.867 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:22:22.867 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-16T15:22:22.867 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:22:22.879 INFO:tasks.workunit.client.0.smithi039.stdout:create new divergent objects 2024-09-16T15:22:22.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-09-16T15:22:22.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-09-16T15:22:22.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-16T15:22:22.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-16T15:22:22.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-09-16T15:22:22.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-09-16T15:22:22.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-09-16T15:22:22.883 INFO:tasks.workunit.client.0.smithi039.stdout:writing divergent objects 2024-09-16T15:22:22.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-09-16T15:22:22.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-16T15:22:22.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-16T15:22:22.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-09-16T15:22:22.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-09-16T15:22:22.885 INFO:tasks.workunit.client.0.smithi039.stdout:remove divergent objects 2024-09-16T15:22:22.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-09-16T15:22:22.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-09-16T15:22:22.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-16T15:22:22.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-09-16T15:22:22.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-09-16T15:22:22.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-16T15:22:22.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-09-16T15:22:22.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-09-16T15:22:22.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-09-16T15:22:22.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-09-16T15:22:22.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-09-16T15:22:32.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-09-16T15:22:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:rados: no process found 2024-09-16T15:22:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-09-16T15:22:32.899 INFO:tasks.workunit.client.0.smithi039.stdout:killing all the osds 2024-09-16T15:22:32.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:22:33.341 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:22:33.341 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:22:08.049092+0000 39'29 56:86 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.341 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:22:28.997923+0000 44'29 57:87 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.341 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2468 0 0 19 12 19 active+clean 2024-09-16T15:22:27.052606+0000 40'31 56:113 [2,1,0] 2 [2,1,0] 2 40'31 2024-09-16T15:22:23.501248+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:37:19.298076+0000 4 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:22:23.335498+0000 43'29 56:95 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:22:21.367145+0000 35'29 56:77 [1,2,0] 1 [1,2,0] 1 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 3 0 0 0 0 2105 0 0 18 12 18 active+clean 2024-09-16T15:22:19.722966+0000 51'30 56:81 [1,2,0] 1 [1,2,0] 1 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-16T15:22:17.394800+0000 43'30 56:104 [1,2,0] 1 [1,2,0] 1 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T03:00:52.797756+0000 3 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1138 0 0 16 12 16 active+clean 2024-09-16T15:22:07.997190+0000 35'28 56:80 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:22:13.724213+0000 40'30 56:83 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-16T15:22:18.581628+0000 42'32 56:104 [2,0,1] 2 [2,0,1] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:50:02.365463+0000 6 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-16T15:22:02.244472+0000 36'28 56:93 [1,0,2] 1 [1,0,2] 1 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:11:18.460354+0000 1 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:22:08.048454+0000 44'30 56:97 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:22:19.240869+0000 40'28 56:87 [0,1,2] 0 [0,1,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T18:28:16.449204+0000 1 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:22:12.130441+0000 41'29 56:94 [1,0,2] 1 [1,0,2] 1 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:37:35.561921+0000 3 0 2024-09-16T15:22:33.342 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-16T15:22:04.583920+0000 41'29 56:90 [2,0,1] 2 [2,0,1] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T20:21:45.960892+0000 0 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-16T15:22:29.192279+0000 33'29 57:94 [1,0,2] 1 [1,0,2] 1 33'29 2024-09-16T15:22:29.192062+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T19:41:43.734315+0000 2 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:22:00.096839+0000 43'29 56:94 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:50:42.273513+0000 2 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:22:27.052605+0000 41'29 56:99 [2,1,0] 2 [2,1,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:32:31.346690+0000 3 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 0 0 0 0 1138 0 0 19 12 19 active+clean 2024-09-16T15:22:29.064856+0000 51'31 57:94 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:22:07.996854+0000 39'29 56:86 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-16T15:22:20.039632+0000 43'30 56:95 [1,0,2] 1 [1,0,2] 1 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T21:07:25.107191+0000 3 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 2 0 0 0 0 1234 0 0 18 12 18 active+clean 2024-09-16T15:22:07.611353+0000 51'30 56:103 [2,0,1] 2 [2,0,1] 2 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T21:10:08.649524+0000 3 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:22:16.229919+0000 33'28 56:99 [0,1,2] 0 [0,1,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:10:01.681706+0000 1 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:22:08.048170+0000 43'30 56:87 [1,2,0] 1 [1,2,0] 1 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:21:57.574840+0000 44'29 56:82 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:22:12.569098+0000 39'29 56:101 [2,0,1] 2 [2,0,1] 2 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:41:15.732286+0000 3 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:22:13.724550+0000 40'31 56:95 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.343 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 520 0 0 15 12 15 active+clean 2024-09-16T15:22:08.049022+0000 31'27 56:84 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.344 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:22:17.394875+0000 44'30 56:92 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.344 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1137 0 0 16 12 16 active+clean 2024-09-16T15:22:19.722439+0000 33'28 56:76 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.344 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:22:21.466449+0000 41'30 56:89 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.344 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 6 0 0 0 0 3338 0 0 21 12 21 active+clean 2024-09-16T15:22:23.276217+0000 52'33 56:110 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:33.344 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:22:33.344 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:22:33.344 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:22:33.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-09-16T15:22:33.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:22:33.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:22:33.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:22:33.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:22:33.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:22:33.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:22:33.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-16T15:22:33.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-09-16T15:22:34.044 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T15:22:34.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-16T15:22:34.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-09-16T15:22:34.793 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-16T15:22:34.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-16T15:22:34.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-09-16T15:22:35.376 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-16T15:22:35.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-16T15:22:35.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-09-16T15:22:35.974 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T15:22:35.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-16T15:22:35.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-09-16T15:22:36.680 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already out. 2024-09-16T15:22:36.695 INFO:tasks.workunit.client.0.smithi039.stdout:bringing up non_divergent 0 2 2024-09-16T15:22:36.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-09-16T15:22:36.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:22:37.108 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 0 0 0 1754 0 0 17 12 17 stale+active+clean 2024-09-16T15:22:08.049092+0000 39'29 56:86 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 0 0 0 1754 0 0 17 12 17 stale+active+clean 2024-09-16T15:22:28.997923+0000 44'29 57:87 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2468 0 0 19 12 19 stale+active+clean 2024-09-16T15:22:27.052606+0000 40'31 56:113 [2,1,0] 2 [2,1,0] 2 40'31 2024-09-16T15:22:23.501248+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:37:19.298076+0000 4 0 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1755 0 0 17 12 17 stale+active+clean 2024-09-16T15:22:23.335498+0000 43'29 56:95 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1755 0 0 17 12 17 stale+active+clean 2024-09-16T15:22:21.367145+0000 35'29 56:77 [1,2,0] 1 [1,2,0] 1 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 3 0 0 0 0 2105 0 0 18 12 18 stale+active+clean 2024-09-16T15:22:19.722966+0000 51'30 56:81 [1,2,0] 1 [1,2,0] 1 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 0 0 0 1851 0 0 18 12 18 stale+active+clean 2024-09-16T15:22:17.394800+0000 43'30 56:104 [1,2,0] 1 [1,2,0] 1 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T03:00:52.797756+0000 3 0 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1138 0 0 16 12 16 stale+active+clean 2024-09-16T15:22:07.997190+0000 35'28 56:80 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 0 0 0 2372 0 0 18 12 18 stale+active+clean 2024-09-16T15:22:13.724213+0000 40'30 56:83 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 stale+active+clean 2024-09-16T15:22:18.581628+0000 42'32 56:104 [2,0,1] 2 [2,0,1] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:50:02.365463+0000 6 0 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 stale+active+clean 2024-09-16T15:22:02.244472+0000 36'28 56:93 [1,0,2] 1 [1,0,2] 1 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:11:18.460354+0000 1 0 2024-09-16T15:22:37.109 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 2372 0 0 18 12 18 stale+active+clean 2024-09-16T15:22:08.048454+0000 44'30 56:97 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 stale+active+clean 2024-09-16T15:22:19.240869+0000 40'28 56:87 [0,1,2] 0 [0,1,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T18:28:16.449204+0000 1 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-16T15:22:12.130441+0000 41'29 56:94 [1,0,2] 1 [1,0,2] 1 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:37:35.561921+0000 3 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 stale+active+clean 2024-09-16T15:22:04.583920+0000 41'29 56:90 [2,0,1] 2 [2,0,1] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T20:21:45.960892+0000 0 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1234 0 0 17 12 17 stale+active+clean 2024-09-16T15:22:29.192279+0000 33'29 57:94 [1,0,2] 1 [1,0,2] 1 33'29 2024-09-16T15:22:29.192062+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T19:41:43.734315+0000 2 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-16T15:22:00.096839+0000 43'29 56:94 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:50:42.273513+0000 2 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-16T15:22:27.052605+0000 41'29 56:99 [2,1,0] 2 [2,1,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:32:31.346690+0000 3 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 0 0 0 0 1138 0 0 19 12 19 stale+active+clean 2024-09-16T15:22:29.064856+0000 51'31 57:94 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 0 0 0 1755 0 0 17 12 17 stale+active+clean 2024-09-16T15:22:07.996854+0000 39'29 56:86 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 0 0 0 1851 0 0 18 12 18 stale+active+clean 2024-09-16T15:22:20.039632+0000 43'30 56:95 [1,0,2] 1 [1,0,2] 1 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T21:07:25.107191+0000 3 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 2 0 0 0 0 1234 0 0 18 12 18 stale+active+clean 2024-09-16T15:22:07.611353+0000 51'30 56:103 [2,0,1] 2 [2,0,1] 2 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T21:10:08.649524+0000 3 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 stale+active+clean 2024-09-16T15:22:16.229919+0000 33'28 56:99 [0,1,2] 0 [0,1,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:10:01.681706+0000 1 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 0 0 0 2372 0 0 18 12 18 stale+active+clean 2024-09-16T15:22:08.048170+0000 43'30 56:87 [1,2,0] 1 [1,2,0] 1 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.110 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1754 0 0 17 12 17 stale+active+clean 2024-09-16T15:21:57.574840+0000 44'29 56:82 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.111 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-16T15:22:12.569098+0000 39'29 56:101 [2,0,1] 2 [2,0,1] 2 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:41:15.732286+0000 3 0 2024-09-16T15:22:37.111 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 0 0 0 2988 0 0 19 12 19 stale+active+clean 2024-09-16T15:22:13.724550+0000 40'31 56:95 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.111 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 520 0 0 15 12 15 stale+active+clean 2024-09-16T15:22:08.049022+0000 31'27 56:84 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.111 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 0 0 0 2371 0 0 18 12 18 stale+active+clean 2024-09-16T15:22:17.394875+0000 44'30 56:92 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.111 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1137 0 0 16 12 16 stale+active+clean 2024-09-16T15:22:19.722439+0000 33'28 56:76 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.111 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2371 0 0 18 12 18 stale+active+clean 2024-09-16T15:22:21.466449+0000 41'30 56:89 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.111 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 6 0 0 0 0 3338 0 0 21 12 21 stale+active+clean 2024-09-16T15:22:23.276217+0000 52'33 56:110 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:22:37.111 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:22:37.111 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:22:37.111 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:22:37.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-16T15:22:37.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-09-16T15:22:37.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:22:37.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:22:37.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:22:37.128 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:22:37.128 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:22:37.128 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:22:37.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:22:37.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:22:37.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:22:37.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:22:37.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:22:37.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:22:37.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:22:37.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:22:37.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:22:37.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:22:37.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:22:37.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-16T15:22:37.130 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:22:37.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:22:37.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:22:37.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-16T15:22:37.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:22:37.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:22:37.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:22:37.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:22:37.169 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:22:37.169+0000 7fdc6f0177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:22:37.171 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:22:37.173+0000 7fdc6f0177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:22:37.173 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:22:37.173+0000 7fdc6f0177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:22:37.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:22:37.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:22:37.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:22:37.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:22:37.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:22:37.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:22:37.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:22:37.544 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:22:37.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:22:37.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:22:37.728 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:22:37.729+0000 7fdc6f0177c0 -1 Falling back to public interface 2024-09-16T15:22:38.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:22:39.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:22:39.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:22:39.105 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:22:39.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:22:39.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:22:39.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:22:39.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:22:40.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:22:40.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:22:40.545 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:22:40.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:22:40.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:22:40.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:22:41.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:22:41.107 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:22:41.105+0000 7fdc6f0177c0 -1 osd.0 50 log_to_monitors true 2024-09-16T15:22:42.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:22:42.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:22:42.051 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:22:42.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:22:42.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:22:42.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:22:42.547 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up out weight 0 up_from 71 up_thru 60 down_at 62 last_clean_interval [5,50) [v2:127.0.0.1:6802/1378971955,v1:127.0.0.1:6803/1378971955] [v2:127.0.0.1:6804/1378971955,v1:127.0.0.1:6805/1378971955] exists,up 39492c66-933f-48d1-bb2d-ba158445ce13 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:22:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:22:42.550 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:22:42.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T15:22:42.552 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:22:42.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:22:42.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-16T15:22:42.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T15:22:42.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:22:42.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:22:42.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:22:42.593 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:22:42.593+0000 7f0f446cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:22:42.595 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:22:42.597+0000 7f0f446cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:22:42.597 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:22:42.597+0000 7f0f446cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:22:42.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T15:22:42.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:22:42.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:22:42.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:22:42.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:22:42.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:22:42.989 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:22:42.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:22:42.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:22:42.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:22:43.152 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:22:43.153+0000 7f0f446cc7c0 -1 Falling back to public interface 2024-09-16T15:22:43.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:22:44.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:22:44.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:22:44.407 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:22:44.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:22:44.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:22:44.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:22:44.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:22:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:22:45.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:22:45.956 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:22:45.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:22:45.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:22:45.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:22:46.374 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:22:46.373+0000 7f0f446cc7c0 -1 osd.2 51 log_to_monitors true 2024-09-16T15:22:46.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:22:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:22:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:22:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:22:47.423 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:22:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:22:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:22:47.911 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up out weight 0 up_from 76 up_thru 58 down_at 65 last_clean_interval [15,51) [v2:127.0.0.1:6810/2494859187,v1:127.0.0.1:6811/2494859187] [v2:127.0.0.1:6812/2494859187,v1:127.0.0.1:6813/2494859187] exists,up d9c568d2-49f6-4872-a744-3ce12eff3568 2024-09-16T15:22:47.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:22:47.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:22:47.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:22:47.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-16T15:22:47.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-09-16T15:22:48.529 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already in. 2024-09-16T15:22:48.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-16T15:22:48.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-09-16T15:22:49.684 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already in. 2024-09-16T15:22:49.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T15:22:49.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-09-16T15:22:49.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:22:49.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:22:49.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:22:49.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T15:22:49.712 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:22:49.712 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:22:49.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:22:49.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:22:49.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:22:49.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T15:22:49.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:22:49.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:22:49.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:22:49.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:22:49.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:22:50.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:22:50.461 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:22:50.461 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:22:50.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:22:50.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:22:50.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:22:50.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=304942678020 2024-09-16T15:22:50.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 304942678020 2024-09-16T15:22:50.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-304942678020' 2024-09-16T15:22:50.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:22:50.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:22:50.923 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T15:22:50.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T15:22:50.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T15:22:50.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T15:22:50.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:22:50.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:22:51.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=326417514499 2024-09-16T15:22:51.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 326417514499 2024-09-16T15:22:51.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-304942678020 2-326417514499' 2024-09-16T15:22:51.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:22:51.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-304942678020 2024-09-16T15:22:51.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:22:51.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:22:51.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-304942678020 2024-09-16T15:22:51.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:22:51.103 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 304942678020 2024-09-16T15:22:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=304942678020 2024-09-16T15:22:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 304942678020' 2024-09-16T15:22:51.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:22:51.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 304942678018 -lt 304942678020 2024-09-16T15:22:51.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:22:52.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:22:52.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:22:53.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 304942678020 -lt 304942678020 2024-09-16T15:22:53.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:22:53.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-326417514499 2024-09-16T15:22:53.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:22:53.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:22:53.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-326417514499 2024-09-16T15:22:53.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:22:53.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=326417514499 2024-09-16T15:22:53.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 326417514499' 2024-09-16T15:22:53.046 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 326417514499 2024-09-16T15:22:53.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:22:53.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 326417514499 -lt 326417514499 2024-09-16T15:22:53.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:22:53.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:22:53.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:22:54.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-16T15:22:54.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:22:54.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:22:54.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:22:54.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:22:54.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:22:54.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:22:54.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:22:54.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:22:54.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:22:54.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:22:54.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:22:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:22:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T15:22:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T15:22:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T15:22:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:22:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T15:22:55.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:22:55.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:22:55.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:22:55.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:22:55.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:22:55.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:22:55.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:22:55.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:22:55.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:22:55.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:22:55.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:22:55.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:22:56.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:22:56.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:22:56.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:22:56.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:22:56.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:22:56.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:22:56.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:22:56.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:22:56.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:22:56.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:22:56.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:22:56.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-16T15:22:56.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:22:56.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T15:22:56.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:22:56.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:22:56.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:22:56.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:22:56.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:22:56.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:22:56.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:22:56.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:22:57.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:22:57.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:22:57.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:22:57.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:22:57.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:22:57.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:22:57.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:22:57.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:22:57.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:22:57.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:22:57.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:22:57.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:22:57.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:22:58.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:22:58.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:22:58.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-16T15:22:58.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:22:58.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T15:22:58.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:22:58.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:22:58.875 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:22:58.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:22:58.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:22:58.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:22:58.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:22:58.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:22:59.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:22:59.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:22:59.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:22:59.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:22:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:22:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:22:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:22:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:22:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:22:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:22:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:22:59.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:22:59.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:00.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:00.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:00.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-16T15:23:00.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:00.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-16T15:23:01.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:01.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:01.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:01.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:01.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:01.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:01.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:01.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:01.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:01.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:01.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:01.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:02.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:02.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:02.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:02.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:02.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:02.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:02.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:02.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:02.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:02.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:02.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:02.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-16T15:23:02.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:02.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-16T15:23:04.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:04.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:04.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:04.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:04.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:04.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:04.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:04.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:04.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:04.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:04.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:04.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:05.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:05.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:05.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:05.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:05.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:05.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:05.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:05.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:05.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:05.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:05.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:05.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-16T15:23:05.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:05.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-16T15:23:09.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:09.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:09.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:09.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:09.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:09.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:09.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:09.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:09.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:09.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:09.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:09.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:10.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:10.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:10.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:10.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:10.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:10.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:10.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:10.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:10.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:10.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:10.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:10.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-16T15:23:10.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:10.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-16T15:23:17.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:17.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:17.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:17.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:17.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:17.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:17.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:17.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:18.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:18.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:18.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:18.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:18.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:18.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:18.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:18.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:18.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:18.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:18.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:18.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-16T15:23:18.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:18.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-16T15:23:26.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:26.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:26.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:26.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:26.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:26.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:26.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:26.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:26.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:26.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:26.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:26.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:27.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:27.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:27.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:27.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:27.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:27.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:27.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:27.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:27.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:27.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:27.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:27.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-16T15:23:27.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T15:23:28.100 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "399711d8-16ae-44f4-8cd0-14c37ad5a466", 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "18.2.4-845-ge4e227b3", 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "e4e227b37858f5109138494e01e5ba1ba7b8ffe8", 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-16T15:23:28.101218+0000", 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-16T15:23:28.101 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.102 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 32 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.103 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.104 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.106 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.107 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.108 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.109 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:23:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-16T15:23:28.111 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:19:47.729148Z", 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:19:47.729148Z", 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 18, 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "reef", 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-16T15:23:28.112 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "reef" 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.113 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-16T15:23:28.114 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 85, 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:19:48.698213+0000", 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:22:51.025206+0000", 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-16T15:22:47.089900+0000", 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-16T15:22:49.097188+0000", 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-16T15:23:28.115 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 9, 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "reef", 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-16T15:23:28.116 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-16T15:21:37.723008+0000", 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "on", 2024-09-16T15:23:28.117 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 32, 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 26, 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 32, 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 32, 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 32, 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "84", 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "32", 2024-09-16T15:23:28.118 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T15:23:28.119 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-16T15:23:28.120 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 1.4900000095367432, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 1.4900000095367432, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 0.67000001668930054, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 1, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 1, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.121 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "39492c66-933f-48d1-bb2d-ba158445ce13", 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 50, 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 71, 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 84, 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 62, 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:23:28.122 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.123 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.124 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.125 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/1378971955", 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/1378971955", 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1378971955", 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1378971955", 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.126 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "0c65ed33-581c-493f-b80f-f44168583347", 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 56, 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 64, 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:23:28.128 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.129 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:23:28.130 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/3128683758", 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/3128683758", 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3128683758", 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3128683758", 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "d9c568d2-49f6-4872-a744-3ce12eff3568", 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:23:28.131 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 51, 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 76, 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 82, 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 65, 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:23:28.132 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:23:28.133 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.134 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:23:28.135 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/2494859187", 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/2494859187", 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2494859187", 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2494859187", 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:23:28.136 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:22:33.846912+0000", 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:20:23.353151+0000", 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:23:28.137 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:22:34.602127+0000", 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:20:58.813712+0000", 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:22:35.183726+0000", 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:23:28.138 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:21:33.931416+0000", 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-16T15:23:28.139 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-16T15:23:28.140 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/1378971955,v1:127.0.0.1:6805/1378971955]", 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:23:28.141 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:23:28.142 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:19:55.631544Z", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/1378971955,v1:127.0.0.1:6803/1378971955]", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1378971955,v1:127.0.0.1:6809/1378971955]", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1378971955,v1:127.0.0.1:6807/1378971955]", 2024-09-16T15:23:28.143 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:23:28.144 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/3128683758,v1:127.0.0.1:6813/3128683758]", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:23:28.145 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:20:31.426289Z", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:23:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/3128683758,v1:127.0.0.1:6811/3128683758]", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3128683758,v1:127.0.0.1:6817/3128683758]", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3128683758,v1:127.0.0.1:6815/3128683758]", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:23:28.147 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/2494859187,v1:127.0.0.1:6813/2494859187]", 2024-09-16T15:23:28.148 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:23:28.149 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:21:05.885129Z", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/2494859187,v1:127.0.0.1:6811/2494859187]", 2024-09-16T15:23:28.150 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2494859187,v1:127.0.0.1:6817/2494859187]", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2494859187,v1:127.0.0.1:6815/2494859187]", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:23:28.151 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [] 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 71 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.152 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 76 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 85, 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:23:28.153 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:23:28.154 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-16T15:23:28.155 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-16T15:23:28.156 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-16T15:23:28.157 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-16T15:23:28.158 INFO:tasks.workunit.client.0.smithi039.stderr:report 1607747812 2024-09-16T15:23:28.158 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:23:28.158 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:23:28.158 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:23:28.158 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:23:28.158 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:23:28.158 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:23:28.158 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.158 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.159 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:23:28.160 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:23:28.162 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-16T15:23:28.163 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.164 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-16T15:23:28.165 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 0, 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-16T15:23:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2" 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-16T15:23:28.167 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 32, 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 32, 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-16T15:23:28.168 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 61852, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 81, 2024-09-16T15:23:28.169 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 81, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:23:28.170 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-16T15:23:28.171 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 553, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 553, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 64, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 64, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 96, 2024-09-16T15:23:28.172 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 84560, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 3028, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314488240, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322035957760, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 3100672, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 809691, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:23:28.173 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 113, 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 113, 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 113000000, 2024-09-16T15:23:28.174 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 113000000 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 96, 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:23:28.175 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 84560, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 3028, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314488240, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322035957760, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 3100672, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 809691, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:23:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 113, 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 113, 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 113000000, 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 113000000 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:23:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 32, 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 61852, 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-16T15:23:28.178 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 81, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 81, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:23:28.179 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:23:28.180 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1220608, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 184566, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.181 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 553, 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 553, 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 64, 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 64, 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 304942678027 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672989 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 326417514506 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 32 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 16, 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 32, 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:23:28.183 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 16, 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 32, 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:23:28.184 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 5, 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:22:36.285950+0000", 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 98, 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 445, 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 608, 2024-09-16T15:23:28.185 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-16T15:23:28.186 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-16T15:23:28.186 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:23:28.186 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:23:28.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T15:23:28.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-09-16T15:23:28.186 INFO:tasks.workunit.client.0.smithi039.stdout:writing non-divergent object existing_4 2024-09-16T15:23:28.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-09-16T15:23:28.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-09-16T15:23:28.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782655+0000 39'29 84:79 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783280+0000 44'29 84:74 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 4 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.782801+0000 40'31 84:88 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.191157+0000 43'29 84:88 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.784157+0000 35'29 84:69 [2,0] 2 [2,0] 2 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783775+0000 41'29 84:89 [2,0] 2 [2,0] 2 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T03:03:15.711645+0000 5 0 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.785432+0000 43'30 84:78 [2,0] 2 [2,0] 2 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.682135+0000 35'28 84:67 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.218245+0000 40'30 84:78 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-16T15:22:50.783701+0000 42'32 84:105 [2,0] 2 [2,0] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:14:36.120841+0000 6 0 2024-09-16T15:23:28.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.219106+0000 36'28 84:88 [0,2] 0 [0,2] 0 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:56:56.198484+0000 1 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.783015+0000 44'30 84:93 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.220337+0000 40'28 84:94 [0,2] 0 [0,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T18:09:39.121709+0000 1 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.217839+0000 41'29 84:82 [0,2] 0 [0,2] 0 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783661+0000 41'29 84:86 [2,0] 2 [2,0] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T20:56:07.416117+0000 0 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.218496+0000 33'29 84:74 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.217858+0000 43'29 84:96 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:44:08.393946+0000 2 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.784415+0000 41'29 84:94 [2,0] 2 [2,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:19:36.607646+0000 3 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782275+0000 41'29 84:69 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782817+0000 39'29 84:78 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.217690+0000 43'30 84:85 [0,2] 0 [0,2] 0 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.219514+0000 33'29 84:74 [0,2] 0 [0,2] 0 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.217655+0000 33'28 84:103 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:50:48.383289+0000 1 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.218291+0000 43'30 84:89 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.567 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782353+0000 44'29 84:78 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.568 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.218801+0000 39'29 84:87 [0,2] 0 [0,2] 0 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.568 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 5 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.221011+0000 40'31 84:91 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.568 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-16T15:22:50.219037+0000 31'27 84:60 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.568 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 4 0 0 2371 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.783330+0000 44'30 84:79 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.568 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:23:06.784930+0000 33'28 85:72 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.568 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 4 0 0 2371 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.784619+0000 41'30 84:82 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.568 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 5 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.218618+0000 44'31 84:102 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:23:28.568 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:23:28.568 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:23:28.568 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:23:28.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-09-16T15:23:28.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T15:23:28.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-09-16T15:23:28.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:23:28.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:23:28.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:23:28.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T15:23:28.760 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:23:28.760 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:23:28.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:23:28.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:23:28.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:23:28.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T15:23:28.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:23:28.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:23:28.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:23:28.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:23:28.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:23:29.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:23:29.273 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:23:29.273 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:23:29.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:23:29.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:23:29.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:23:29.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=304942678030 2024-09-16T15:23:29.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 304942678030 2024-09-16T15:23:29.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-304942678030' 2024-09-16T15:23:29.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:23:29.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:23:29.572 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T15:23:29.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T15:23:29.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T15:23:29.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T15:23:29.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:23:29.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:23:29.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=326417514509 2024-09-16T15:23:29.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 326417514509 2024-09-16T15:23:29.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-304942678030 2-326417514509' 2024-09-16T15:23:29.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:23:29.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:23:29.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-304942678030 2024-09-16T15:23:29.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:23:29.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-304942678030 2024-09-16T15:23:29.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:23:29.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=304942678030 2024-09-16T15:23:29.741 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 304942678030 2024-09-16T15:23:29.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 304942678030' 2024-09-16T15:23:29.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:23:30.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 304942678028 -lt 304942678030 2024-09-16T15:23:30.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:23:31.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:23:31.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:23:31.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 304942678030 -lt 304942678030 2024-09-16T15:23:31.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:23:31.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-326417514509 2024-09-16T15:23:31.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:23:31.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:23:31.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-326417514509 2024-09-16T15:23:31.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:23:31.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=326417514509 2024-09-16T15:23:31.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 326417514509' 2024-09-16T15:23:31.878 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 326417514509 2024-09-16T15:23:31.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:23:32.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 326417514509 -lt 326417514509 2024-09-16T15:23:32.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:23:32.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:32.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:32.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-16T15:23:32.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:32.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:32.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:32.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:32.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:32.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:32.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:33.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:33.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:33.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:33.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:33.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:33.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T15:23:33.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T15:23:33.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T15:23:33.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:33.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T15:23:33.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:33.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:33.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:33.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:33.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:33.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:33.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:33.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:34.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:34.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:34.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:34.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:34.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:34.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:34.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:35.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:35.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:35.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-16T15:23:35.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:35.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T15:23:35.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:35.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:35.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:35.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:35.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:35.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:35.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:35.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:36.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:36.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:36.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:36.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:36.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:36.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:37.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:37.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:37.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-16T15:23:37.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:37.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T15:23:37.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:37.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:37.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:37.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:37.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:37.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:37.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:37.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:38.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:38.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:38.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:38.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:38.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:38.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:38.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:38.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:38.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:38.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:38.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:38.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:39.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:39.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:39.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-16T15:23:39.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:39.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-16T15:23:39.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:39.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:39.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:39.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:39.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:39.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:39.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:39.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:40.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:40.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:40.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:40.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:40.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:40.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:40.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:40.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:40.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:40.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:40.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:40.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:40.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:41.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:41.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:41.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-16T15:23:41.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:41.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-16T15:23:43.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:43.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:43.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:43.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:43.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:43.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:43.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:43.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:43.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:43.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:43.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:43.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:44.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:44.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:44.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:44.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:44.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:44.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:44.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:44.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:44.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:44.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:44.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:44.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-16T15:23:44.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:44.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-16T15:23:47.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:47.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:47.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:47.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:47.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:47.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:47.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:47.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:48.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:48.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:48.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:48.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:48.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:48.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:48.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:48.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:48.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:48.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:48.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:48.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:48.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:49.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:49.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:49.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-16T15:23:49.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:49.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-16T15:23:55.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:23:55.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:23:55.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:23:55.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:23:55.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:23:55.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:23:55.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:23:55.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:23:56.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:23:56.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:23:56.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:23:56.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:23:56.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:23:56.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:23:56.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:23:56.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:23:56.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:23:56.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:23:56.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:23:56.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:23:56.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:23:57.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:23:57.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:23:57.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-16T15:23:57.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:23:57.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-16T15:24:04.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:24:04.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:24:04.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:24:04.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:24:04.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:24:04.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:24:04.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:24:04.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:24:05.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:24:05.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:24:05.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:24:05.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:24:05.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-16T15:24:05.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:24:05.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:24:05.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:24:05.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:24:05.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:24:05.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:24:05.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:24:05.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:24:06.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:24:06.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:24:06.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-16T15:24:06.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T15:24:06.893 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:24:06.893 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "399711d8-16ae-44f4-8cd0-14c37ad5a466", 2024-09-16T15:24:06.893 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "18.2.4-845-ge4e227b3", 2024-09-16T15:24:06.893 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "e4e227b37858f5109138494e01e5ba1ba7b8ffe8", 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-16T15:24:06.894192+0000", 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.894 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.895 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 64 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.896 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.2 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.3 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.4 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.5 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.6 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.897 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.7 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.8 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.9 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.a is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.b is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.898 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.c is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.d is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.e is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.f is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.10 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.899 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.11 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.12 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.13 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.14 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.15 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.900 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.16 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.17 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.18 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.19 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1a is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.901 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1b is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1c is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1d is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1e is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1f is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-16T15:24:06.902 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:19:47.729148Z", 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:19:47.729148Z", 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 18, 2024-09-16T15:24:06.904 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "reef", 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-16T15:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: "reef" 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-16T15:24:06.906 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-16T15:24:06.907 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 85, 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:19:48.698213+0000", 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:22:51.025206+0000", 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-16T15:22:47.089900+0000", 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-16T15:22:49.097188+0000", 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-16T15:24:06.908 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 9, 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-16T15:24:06.909 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "reef", 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-16T15:21:37.723008+0000", 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-16T15:24:06.910 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "on", 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 32, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 26, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 32, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 32, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 32, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-16T15:24:06.911 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "84", 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "32", 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-16T15:24:06.912 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-16T15:24:06.913 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-16T15:24:06.914 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 1.4900000095367432, 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 1.4900000095367432, 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 0.67000001668930054, 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 1, 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 1, 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:24:06.915 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "39492c66-933f-48d1-bb2d-ba158445ce13", 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 50, 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 71, 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 84, 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 62, 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-16T15:24:06.916 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:24:06.917 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:24:06.918 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.919 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1378971955 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/1378971955", 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/1378971955", 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1378971955", 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1378971955", 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:24:06.920 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "0c65ed33-581c-493f-b80f-f44168583347", 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 56, 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 64, 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.921 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:24:06.922 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.923 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.924 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3128683758 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/3128683758", 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/3128683758", 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3128683758", 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3128683758", 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.925 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "d9c568d2-49f6-4872-a744-3ce12eff3568", 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 51, 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 76, 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 82, 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 65, 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.926 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.927 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.928 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:24:06.929 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2494859187 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/2494859187", 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/2494859187", 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2494859187", 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2494859187", 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:24:06.930 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:22:33.846912+0000", 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:20:23.353151+0000", 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:24:06.931 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:22:34.602127+0000", 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:20:58.813712+0000", 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:22:35.183726+0000", 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:21:33.931416+0000", 2024-09-16T15:24:06.932 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-16T15:24:06.933 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.934 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/1378971955,v1:127.0.0.1:6805/1378971955]", 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:24:06.935 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:24:06.936 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:19:55.631544Z", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/1378971955,v1:127.0.0.1:6803/1378971955]", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1378971955,v1:127.0.0.1:6809/1378971955]", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1378971955,v1:127.0.0.1:6807/1378971955]", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:24:06.937 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/3128683758,v1:127.0.0.1:6813/3128683758]", 2024-09-16T15:24:06.938 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:20:31.426289Z", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/3128683758,v1:127.0.0.1:6811/3128683758]", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:24:06.940 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3128683758,v1:127.0.0.1:6817/3128683758]", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3128683758,v1:127.0.0.1:6815/3128683758]", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.941 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/2494859187,v1:127.0.0.1:6813/2494859187]", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-16T15:24:06.942 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:21:05.885129Z", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:24:06.943 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/2494859187,v1:127.0.0.1:6811/2494859187]", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2494859187,v1:127.0.0.1:6817/2494859187]", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2494859187,v1:127.0.0.1:6815/2494859187]", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:24:06.944 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [] 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 71 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.945 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 76 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 85, 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.946 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.947 INFO:tasks.workunit.client.0.smithi039.stderr:report 3618754678 2024-09-16T15:24:06.947 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:24:06.947 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-16T15:24:06.947 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:24:06.947 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.947 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.947 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-16T15:24:06.948 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-16T15:24:06.949 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-16T15:24:06.950 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:24:06.951 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:24:06.952 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:24:06.953 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.954 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:24:06.955 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-16T15:24:06.956 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.957 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-16T15:24:06.958 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 0, 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-16T15:24:06.959 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2" 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-16T15:24:06.960 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 32, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 32, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-16T15:24:06.961 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62202, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 82, 2024-09-16T15:24:06.962 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 82, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:24:06.963 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-16T15:24:06.964 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 554, 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 554, 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 64, 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 64, 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:24:06.965 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 96, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 84560, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 3028, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314488240, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322035957760, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:24:06.966 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 3100672, 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 810391, 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:24:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 113, 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 113, 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 113000000, 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 113000000 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 96, 2024-09-16T15:24:06.968 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 84560, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 3028, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314488240, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322035957760, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 3100672, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 810391, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:24:06.969 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 113, 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 113, 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 113000000, 2024-09-16T15:24:06.970 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 113000000 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 32, 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62202, 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:24:06.971 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 82, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 82, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:24:06.973 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1220608, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 185266, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.974 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 554, 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 554, 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 64, 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 64, 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 304942678037 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672989 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.975 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 326417514516 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 32 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 16, 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 32, 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:24:06.976 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 16, 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 32, 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-16T15:24:06.977 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 7, 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:23:48.313178+0000", 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 118, 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 513, 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:24:06.978 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 696, 2024-09-16T15:24:06.979 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-16T15:24:06.979 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-16T15:24:06.979 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:24:06.979 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:24:06.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T15:24:06.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-09-16T15:24:06.979 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery 2024-09-16T15:24:06.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:24:07.309 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:24:07.309 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782655+0000 39'29 84:79 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.309 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783280+0000 44'29 84:74 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.309 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 4 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.782801+0000 40'31 84:88 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.309 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.191157+0000 43'29 84:88 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.309 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.784157+0000 35'29 84:69 [2,0] 2 [2,0] 2 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.309 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783775+0000 41'29 84:89 [2,0] 2 [2,0] 2 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T03:03:15.711645+0000 5 0 2024-09-16T15:24:07.309 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.785432+0000 43'30 84:78 [2,0] 2 [2,0] 2 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.309 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.682135+0000 35'28 84:67 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.218245+0000 40'30 84:78 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-16T15:22:50.783701+0000 42'32 84:105 [2,0] 2 [2,0] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:14:36.120841+0000 6 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.219106+0000 36'28 84:88 [0,2] 0 [0,2] 0 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:56:56.198484+0000 1 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.783015+0000 44'30 84:93 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.220337+0000 40'28 84:94 [0,2] 0 [0,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T18:09:39.121709+0000 1 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.217839+0000 41'29 84:82 [0,2] 0 [0,2] 0 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783661+0000 41'29 84:86 [2,0] 2 [2,0] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T20:56:07.416117+0000 0 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.218496+0000 33'29 84:74 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.217858+0000 43'29 84:96 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:44:08.393946+0000 2 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.784415+0000 41'29 84:94 [2,0] 2 [2,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:19:36.607646+0000 3 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 0 3 0 0 2105 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.782275+0000 85'30 85:70 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782817+0000 39'29 84:78 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.217690+0000 43'30 84:85 [0,2] 0 [0,2] 0 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.310 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.219514+0000 33'29 84:74 [0,2] 0 [0,2] 0 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.217655+0000 33'28 84:103 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:50:48.383289+0000 1 0 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.218291+0000 43'30 84:89 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782353+0000 44'29 84:78 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.218801+0000 39'29 84:87 [0,2] 0 [0,2] 0 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 5 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.221011+0000 40'31 84:91 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-16T15:22:50.219037+0000 31'27 84:60 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 4 0 0 2371 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.783330+0000 44'30 84:79 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:23:06.784930+0000 33'28 85:72 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 4 0 0 2371 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.784619+0000 41'30 84:82 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 5 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.218618+0000 44'31 84:102 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:24:07.311 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:24:07.312 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:24:07.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-16T15:24:07.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-09-16T15:24:07.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T15:24:07.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T15:24:07.325 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:24:07.325 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:07.325 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:07.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.0.asok 2024-09-16T15:24:07.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-16T15:24:07.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.0.asok set_recovery_delay 100000 2024-09-16T15:24:07.436 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:24:07.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-16T15:24:07.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-09-16T15:24:07.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T15:24:07.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T15:24:07.451 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:24:07.451 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:07.451 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:07.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.2.asok 2024-09-16T15:24:07.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-16T15:24:07.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.2.asok set_recovery_delay 100000 2024-09-16T15:24:07.555 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:24:07.567 INFO:tasks.workunit.client.0.smithi039.stdout:revive divergent 1 2024-09-16T15:24:07.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-09-16T15:24:07.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:24:08.011 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:24:08.012 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782655+0000 39'29 84:79 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.012 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783280+0000 44'29 84:74 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.012 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 4 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.782801+0000 40'31 84:88 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.012 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.191157+0000 43'29 84:88 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.012 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.784157+0000 35'29 84:69 [2,0] 2 [2,0] 2 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.012 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783775+0000 41'29 84:89 [2,0] 2 [2,0] 2 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T03:03:15.711645+0000 5 0 2024-09-16T15:24:08.012 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.785432+0000 43'30 84:78 [2,0] 2 [2,0] 2 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.012 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.682135+0000 35'28 84:67 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.012 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.218245+0000 40'30 84:78 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.012 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-16T15:22:50.783701+0000 42'32 84:105 [2,0] 2 [2,0] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:14:36.120841+0000 6 0 2024-09-16T15:24:08.013 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.219106+0000 36'28 84:88 [0,2] 0 [0,2] 0 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:56:56.198484+0000 1 0 2024-09-16T15:24:08.013 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.783015+0000 44'30 84:93 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.013 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.220337+0000 40'28 84:94 [0,2] 0 [0,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T18:09:39.121709+0000 1 0 2024-09-16T15:24:08.013 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.217839+0000 41'29 84:82 [0,2] 0 [0,2] 0 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.013 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783661+0000 41'29 84:86 [2,0] 2 [2,0] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T20:56:07.416117+0000 0 0 2024-09-16T15:24:08.013 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.218496+0000 33'29 84:74 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.013 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.217858+0000 43'29 84:96 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:44:08.393946+0000 2 0 2024-09-16T15:24:08.013 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.784415+0000 41'29 84:94 [2,0] 2 [2,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:19:36.607646+0000 3 0 2024-09-16T15:24:08.013 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 0 3 0 0 2105 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.782275+0000 85'30 85:70 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.013 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782817+0000 39'29 84:78 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.013 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.217690+0000 43'30 84:85 [0,2] 0 [0,2] 0 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.219514+0000 33'29 84:74 [0,2] 0 [0,2] 0 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.217655+0000 33'28 84:103 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:50:48.383289+0000 1 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.218291+0000 43'30 84:89 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782353+0000 44'29 84:78 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.218801+0000 39'29 84:87 [0,2] 0 [0,2] 0 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 5 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.221011+0000 40'31 84:91 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-16T15:22:50.219037+0000 31'27 84:60 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 4 0 0 2371 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.783330+0000 44'30 84:79 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:23:06.784930+0000 33'28 85:72 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 4 0 0 2371 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.784619+0000 41'30 84:82 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 5 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.218618+0000 44'31 84:102 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:24:08.014 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:24:08.015 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:24:08.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-09-16T15:24:08.964 INFO:tasks.workunit.client.0.smithi039.stderr:noup is set 2024-09-16T15:24:08.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:24:08.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:24:08.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:24:08.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:24:08.990 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:24:08.990 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:08.990 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:08.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:24:08.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:24:08.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:24:08.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:24:08.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:24:08.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:24:08.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:24:08.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:24:08.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:24:08.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:24:08.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:24:08.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:24:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:24:08.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:24:08.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:24:08.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-16T15:24:08.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:24:08.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:24:08.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:24:08.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:24:09.040 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:09.043+0000 7fe673ff07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:09.062 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:09.063+0000 7fe673ff07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:09.064 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:09.067+0000 7fe673ff07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:09.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-16T15:24:09.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-09-16T15:24:09.601 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:09.603+0000 7fe673ff07c0 -1 Falling back to public interface 2024-09-16T15:24:12.855 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:12.859+0000 7fe673ff07c0 -1 osd.1 61 log_to_monitors true 2024-09-16T15:24:14.477 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery divergent 2024-09-16T15:24:14.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-09-16T15:24:14.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:24:14.922 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:24:14.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782655+0000 39'29 86:81 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783280+0000 44'29 86:76 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 4 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.782801+0000 40'31 86:90 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.191157+0000 43'29 86:90 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.784157+0000 35'29 86:71 [2,0] 2 [2,0] 2 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783775+0000 41'29 86:91 [2,0] 2 [2,0] 2 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T03:03:15.711645+0000 5 0 2024-09-16T15:24:14.922 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.785432+0000 43'30 86:80 [2,0] 2 [2,0] 2 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.682135+0000 35'28 86:69 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.218245+0000 40'30 86:80 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-16T15:22:50.783701+0000 42'32 86:107 [2,0] 2 [2,0] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:14:36.120841+0000 6 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.219106+0000 36'28 86:90 [0,2] 0 [0,2] 0 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:56:56.198484+0000 1 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.783015+0000 44'30 86:95 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.220337+0000 40'28 86:96 [0,2] 0 [0,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T18:09:39.121709+0000 1 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.217839+0000 41'29 86:84 [0,2] 0 [0,2] 0 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783661+0000 41'29 86:88 [2,0] 2 [2,0] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T20:56:07.416117+0000 0 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.218496+0000 33'29 86:76 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.217858+0000 43'29 86:98 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:44:08.393946+0000 2 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.784415+0000 41'29 86:96 [2,0] 2 [2,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:19:36.607646+0000 3 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 0 3 0 0 2105 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.782275+0000 85'30 86:72 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782817+0000 39'29 86:80 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.217690+0000 43'30 86:87 [0,2] 0 [0,2] 0 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.923 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.219514+0000 33'29 86:76 [0,2] 0 [0,2] 0 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.217655+0000 33'28 86:105 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:50:48.383289+0000 1 0 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.218291+0000 43'30 86:91 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782353+0000 44'29 86:80 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.218801+0000 39'29 86:89 [0,2] 0 [0,2] 0 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 5 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.221011+0000 40'31 86:93 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-16T15:22:50.219037+0000 31'27 86:62 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 4 0 0 2371 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.783330+0000 44'30 86:81 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:23:06.784930+0000 33'28 86:74 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 4 0 0 2371 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.784619+0000 41'30 86:84 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 5 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.218618+0000 44'31 86:104 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:24:14.924 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:24:14.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-09-16T15:24:14.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T15:24:14.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T15:24:14.941 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:24:14.941 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:14.941 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:14.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.1.asok 2024-09-16T15:24:14.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-09-16T15:24:14.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.1.asok set_recovery_delay 100000 2024-09-16T15:24:15.059 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:24:15.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-09-16T15:24:15.846 INFO:tasks.workunit.client.0.smithi039.stderr:noup is unset 2024-09-16T15:24:15.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-09-16T15:24:15.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:24:15.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:24:15.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:24:15.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:24:15.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:24:15.868 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:24:15.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:24:15.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:24:15.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:24:16.707 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 71 up_thru 84 down_at 62 last_clean_interval [5,50) [v2:127.0.0.1:6802/1378971955,v1:127.0.0.1:6803/1378971955] [v2:127.0.0.1:6804/1378971955,v1:127.0.0.1:6805/1378971955] exists,up 39492c66-933f-48d1-bb2d-ba158445ce13 2024-09-16T15:24:16.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:24:16.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:24:16.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:24:16.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-09-16T15:24:16.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:24:16.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:24:16.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:24:16.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:24:16.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:24:16.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:24:16.710 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:24:16.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:24:16.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:24:17.226 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 64 last_clean_interval [10,61) [v2:127.0.0.1:6818/4287159237,v1:127.0.0.1:6819/4287159237] [v2:127.0.0.1:6820/4287159237,v1:127.0.0.1:6821/4287159237] exists,up 0c65ed33-581c-493f-b80f-f44168583347 2024-09-16T15:24:17.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:24:17.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:24:17.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:24:17.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-09-16T15:24:17.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:24:17.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:24:17.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:24:17.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:24:17.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:24:17.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:24:17.229 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:24:17.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:24:17.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:24:17.699 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 76 up_thru 90 down_at 65 last_clean_interval [15,51) [v2:127.0.0.1:6810/2494859187,v1:127.0.0.1:6811/2494859187] [v2:127.0.0.1:6812/2494859187,v1:127.0.0.1:6813/2494859187] exists,up d9c568d2-49f6-4872-a744-3ce12eff3568 2024-09-16T15:24:17.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:24:17.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:24:17.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:24:17.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:24:18.128 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:24:18.128 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782655+0000 39'29 86:81 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.128 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783280+0000 44'29 86:76 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.128 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 4 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.782801+0000 40'31 86:90 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.128 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.191157+0000 43'29 86:90 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.128 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.784157+0000 35'29 86:71 [2,0] 2 [2,0] 2 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.128 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783775+0000 41'29 86:91 [2,0] 2 [2,0] 2 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T03:03:15.711645+0000 5 0 2024-09-16T15:24:18.129 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.785432+0000 43'30 86:80 [2,0] 2 [2,0] 2 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.129 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.682135+0000 35'28 86:69 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.129 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.218245+0000 40'30 86:80 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.129 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-16T15:22:50.783701+0000 42'32 86:107 [2,0] 2 [2,0] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:14:36.120841+0000 6 0 2024-09-16T15:24:18.129 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.219106+0000 36'28 86:90 [0,2] 0 [0,2] 0 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:56:56.198484+0000 1 0 2024-09-16T15:24:18.129 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.783015+0000 44'30 86:95 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.129 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.220337+0000 40'28 86:96 [0,2] 0 [0,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T18:09:39.121709+0000 1 0 2024-09-16T15:24:18.129 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.217839+0000 41'29 86:84 [0,2] 0 [0,2] 0 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.129 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.783661+0000 41'29 86:88 [2,0] 2 [2,0] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T20:56:07.416117+0000 0 0 2024-09-16T15:24:18.129 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 2 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.218496+0000 33'29 86:76 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.217858+0000 43'29 86:98 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:44:08.393946+0000 2 0 2024-09-16T15:24:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.784415+0000 41'29 86:96 [2,0] 2 [2,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T17:19:36.607646+0000 3 0 2024-09-16T15:24:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 0 3 0 0 2105 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.782275+0000 85'30 86:72 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782817+0000 39'29 86:80 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.217690+0000 43'30 86:87 [0,2] 0 [0,2] 0 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.219514+0000 33'29 86:76 [0,2] 0 [0,2] 0 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:22:50.217655+0000 33'28 86:105 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:50:48.383289+0000 1 0 2024-09-16T15:24:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.218291+0000 43'30 86:91 [0,2] 0 [0,2] 0 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.782353+0000 44'29 86:80 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.130 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-16T15:22:50.218801+0000 39'29 86:89 [0,2] 0 [0,2] 0 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.131 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 5 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.221011+0000 40'31 86:93 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.131 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-16T15:22:50.219037+0000 31'27 86:62 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.131 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 4 0 0 2371 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.783330+0000 44'30 86:81 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.131 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-16T15:23:06.784930+0000 33'28 86:74 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.131 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 4 0 0 2371 0 0 18 12 18 active+undersized+degraded 2024-09-16T15:22:50.784619+0000 41'30 86:84 [2,0] 2 [2,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.131 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 5 0 0 2988 0 0 19 12 19 active+undersized+degraded 2024-09-16T15:22:50.218618+0000 44'31 86:104 [0,2] 0 [0,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.131 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:24:18.131 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:24:18.132 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:24:18.144 INFO:tasks.workunit.client.0.smithi039.stdout:wait for peering 2024-09-16T15:24:18.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-09-16T15:24:18.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:24:18.548 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:24:18.548 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.785104+0000 39'29 91:94 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.548 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.789974+0000 44'29 91:102 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.548 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:24:16.780780+0000 40'31 91:103 [2,1,0] 2 [2,1,0] 2 40'31 2024-09-16T15:22:23.501248+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:16.785579+0000 43'29 91:105 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:16.790240+0000 35'29 91:85 [1,2,0] 1 [1,2,0] 1 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:33:03.930187+0000 3 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 3 1 1 0 0 1755 0 0 17 12 17 active+recovery_wait+degraded 2024-09-16T15:24:16.786301+0000 41'29 91:88 [1,2,0] 1 [1,2,0] 1 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.782895+0000 43'30 91:108 [1,2,0] 1 [1,2,0] 1 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1138 0 0 16 12 16 active+clean 2024-09-16T15:24:16.782508+0000 35'28 91:109 [1,2,0] 1 [1,2,0] 1 35'28 2024-09-16T15:22:32.069055+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.785090+0000 40'30 91:84 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-16T15:24:16.783396+0000 42'32 91:120 [2,0,1] 2 [2,0,1] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T19:46:22.853936+0000 6 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-16T15:24:16.785544+0000 36'28 91:87 [1,0,2] 1 [1,0,2] 1 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T00:26:01.879701+0000 1 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.779471+0000 44'30 91:99 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:24:17.299191+0000 40'28 92:110 [0,1,2] 0 [0,1,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T23:14:15.382634+0000 1 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:16.782044+0000 41'29 91:93 [1,0,2] 1 [1,0,2] 1 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-16T15:24:16.785730+0000 41'29 91:101 [2,0,1] 2 [2,0,1] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:54:57.765086+0000 0 0 2024-09-16T15:24:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.783453+0000 33'29 91:109 [1,0,2] 1 [1,0,2] 1 33'29 2024-09-16T15:22:29.192062+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:16.788943+0000 43'29 91:95 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:00:15.112885+0000 2 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:16.781114+0000 41'29 91:109 [2,1,0] 2 [2,1,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:40:56.328702+0000 3 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 1 1 0 0 2105 0 0 18 12 18 active+recovery_wait+degraded 2024-09-16T15:24:16.785001+0000 85'30 91:107 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:16.783202+0000 39'29 91:93 [2,0,1] 2 [2,0,1] 2 39'29 2024-09-16T15:22:32.471614+0000 39'29 2024-09-16T15:22:32.471614+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:17.221479+0000 43'30 92:101 [0,2,1] 0 [0,2,1] 0 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 1 0 0 1755 0 0 17 12 17 active+recovery_wait+degraded 2024-09-16T15:24:17.299479+0000 33'29 92:89 [0,1,2] 0 [0,1,2] 0 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:24:17.180759+0000 33'28 92:119 [0,1,2] 0 [0,1,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:11:37.102392+0000 1 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.786041+0000 43'30 91:88 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.783014+0000 44'29 91:93 [2,0,1] 2 [2,0,1] 2 44'29 2024-09-16T15:22:31.091358+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.690375+0000 39'29 92:84 [1,0,2] 1 [1,0,2] 1 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:24:16.779750+0000 40'31 91:96 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 520 0 0 15 12 15 active+clean 2024-09-16T15:24:17.221460+0000 31'27 92:76 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:24:16.784154+0000 44'30 91:94 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.550 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1137 0 0 16 12 16 active+clean 2024-09-16T15:24:16.790676+0000 33'28 91:82 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.551 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:24:16.785338+0000 41'30 91:97 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.551 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 1 1 0 0 2988 0 0 19 12 19 active+recovering+degraded 2024-09-16T15:24:16.789686+0000 44'31 91:117 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:18.551 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:24:18.551 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:24:18.551 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:24:18.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-09-16T15:24:18.717 INFO:tasks.workunit.client.0.smithi039.stdout:killing divergent 1 2024-09-16T15:24:18.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-09-16T15:24:18.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:24:19.125 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:24:19.125 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.785104+0000 39'29 91:94 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.125 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.789974+0000 44'29 91:102 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.125 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:24:16.780780+0000 40'31 91:103 [2,1,0] 2 [2,1,0] 2 40'31 2024-09-16T15:22:23.501248+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.125 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:16.785579+0000 43'29 91:105 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.125 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:16.790240+0000 35'29 91:85 [1,2,0] 1 [1,2,0] 1 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:33:03.930187+0000 3 0 2024-09-16T15:24:19.125 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 3 1 1 0 0 1755 0 0 17 12 17 active+recovery_wait+degraded 2024-09-16T15:24:16.786301+0000 41'29 91:88 [1,2,0] 1 [1,2,0] 1 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.782895+0000 43'30 91:108 [1,2,0] 1 [1,2,0] 1 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1138 0 0 16 12 16 active+clean 2024-09-16T15:24:16.782508+0000 35'28 91:109 [1,2,0] 1 [1,2,0] 1 35'28 2024-09-16T15:22:32.069055+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.785090+0000 40'30 91:84 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-16T15:24:16.783396+0000 42'32 91:120 [2,0,1] 2 [2,0,1] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T19:46:22.853936+0000 6 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-16T15:24:16.785544+0000 36'28 91:87 [1,0,2] 1 [1,0,2] 1 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T00:26:01.879701+0000 1 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.779471+0000 44'30 91:99 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:24:17.299191+0000 40'28 92:110 [0,1,2] 0 [0,1,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T23:14:15.382634+0000 1 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:16.782044+0000 41'29 91:93 [1,0,2] 1 [1,0,2] 1 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-16T15:24:16.785730+0000 41'29 91:101 [2,0,1] 2 [2,0,1] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:54:57.765086+0000 0 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.783453+0000 33'29 91:109 [1,0,2] 1 [1,0,2] 1 33'29 2024-09-16T15:22:29.192062+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:16.788943+0000 43'29 91:95 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:00:15.112885+0000 2 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:16.781114+0000 41'29 91:109 [2,1,0] 2 [2,1,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:40:56.328702+0000 3 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 1 1 0 0 2105 0 0 18 12 18 active+recovery_wait+degraded 2024-09-16T15:24:16.785001+0000 85'30 91:107 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.126 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:16.783202+0000 39'29 91:93 [2,0,1] 2 [2,0,1] 2 39'29 2024-09-16T15:22:32.471614+0000 39'29 2024-09-16T15:22:32.471614+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:17.221479+0000 43'30 92:101 [0,2,1] 0 [0,2,1] 0 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 1 0 0 1755 0 0 17 12 17 active+recovery_wait+degraded 2024-09-16T15:24:17.299479+0000 33'29 92:89 [0,1,2] 0 [0,1,2] 0 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:24:17.180759+0000 33'28 92:119 [0,1,2] 0 [0,1,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:11:37.102392+0000 1 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.786041+0000 43'30 91:88 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.783014+0000 44'29 91:93 [2,0,1] 2 [2,0,1] 2 44'29 2024-09-16T15:22:31.091358+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.690375+0000 39'29 92:84 [1,0,2] 1 [1,0,2] 1 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:24:16.779750+0000 40'31 91:96 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 520 0 0 15 12 15 active+clean 2024-09-16T15:24:17.221460+0000 31'27 92:76 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:24:16.784154+0000 44'30 91:94 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1137 0 0 16 12 16 active+clean 2024-09-16T15:24:16.790676+0000 33'28 91:82 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:24:16.785338+0000 41'30 91:97 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 1 1 0 0 2988 0 0 19 12 19 active+recovering+degraded 2024-09-16T15:24:16.789686+0000 44'31 91:117 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:24:19.127 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:24:19.128 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:24:19.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-09-16T15:24:19.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:24:19.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:24:19.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:24:19.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:24:19.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:24:19.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:24:19.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.27113.out 2024-09-16T15:24:19.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27113.out 2024-09-16T15:24:19.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-16T15:24:19.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T15:24:19.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T15:24:19.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T15:24:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-16T15:24:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27113.out 2024-09-16T15:24:20.536 INFO:tasks.workunit.client.0.smithi039.stderr:Exporting 1.1b info 1.1b( v 44'31 lc 0'0 (30'12,44'31] local-lis/les=90/91 n=5 ec=32/22 lis/c=90/51 les/c/f=91/52/0 sis=90) 2024-09-16T15:24:20.537 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-16T15:24:20.557 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-16T15:24:20.558 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-16T15:24:20.564 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-16T15:24:20.564 INFO:tasks.workunit.client.0.smithi039.stderr:Export successful 2024-09-16T15:24:20.565 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-16T15:24:20.565 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-16T15:24:20.565 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-09-16T15:24:20.695 INFO:tasks.workunit.client.0.smithi039.stderr:Remove successful 2024-09-16T15:24:21.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.27113.out 2024-09-16T15:24:21.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-16T15:24:21.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T15:24:21.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T15:24:21.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T15:24:21.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-16T15:24:21.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.27113.out 2024-09-16T15:24:22.101 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e92 pg_nums {1={22=1,32=32}} deleted_pools ) 2024-09-16T15:24:22.102 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.1b 2024-09-16T15:24:22.102 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-16T15:24:22.102 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:24:22.156 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-16T15:24:22.156 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:24:22.181 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-16T15:24:22.181 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:24:22.206 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-16T15:24:22.206 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T15:24:22.231 INFO:tasks.workunit.client.0.smithi039.stdout:write_pg epoch 91 info 1.1b( v 44'31 lc 0'0 (30'12,44'31] local-lis/les=90/91 n=5 ec=32/22 lis/c=90/51 les/c/f=91/52/0 sis=90) 2024-09-16T15:24:22.256 INFO:tasks.workunit.client.0.smithi039.stdout:Import successful 2024-09-16T15:24:22.797 INFO:tasks.workunit.client.0.smithi039.stdout:reviving divergent 1 2024-09-16T15:24:22.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-09-16T15:24:22.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:24:23.216 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:24:23.216 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.785104+0000 39'29 91:94 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.789974+0000 44'29 91:102 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:24:16.780780+0000 40'31 91:103 [2,1,0] 2 [2,1,0] 2 40'31 2024-09-16T15:22:23.501248+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:16.785579+0000 43'29 91:105 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:16.790240+0000 35'29 91:85 [1,2,0] 1 [1,2,0] 1 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:33:03.930187+0000 3 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 3 1 1 0 0 1755 0 0 17 12 17 active+recovery_wait+degraded 2024-09-16T15:24:16.786301+0000 41'29 91:88 [1,2,0] 1 [1,2,0] 1 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.782895+0000 43'30 91:108 [1,2,0] 1 [1,2,0] 1 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1138 0 0 16 12 16 active+clean 2024-09-16T15:24:16.782508+0000 35'28 91:109 [1,2,0] 1 [1,2,0] 1 35'28 2024-09-16T15:22:32.069055+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.785090+0000 40'30 91:84 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-16T15:24:16.783396+0000 42'32 91:120 [2,0,1] 2 [2,0,1] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T19:46:22.853936+0000 6 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-16T15:24:16.785544+0000 36'28 91:87 [1,0,2] 1 [1,0,2] 1 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T00:26:01.879701+0000 1 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.779471+0000 44'30 91:99 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:24:17.299191+0000 40'28 92:110 [0,1,2] 0 [0,1,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T23:14:15.382634+0000 1 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:16.782044+0000 41'29 91:93 [1,0,2] 1 [1,0,2] 1 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-16T15:24:16.785730+0000 41'29 91:101 [2,0,1] 2 [2,0,1] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:54:57.765086+0000 0 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.783453+0000 33'29 91:109 [1,0,2] 1 [1,0,2] 1 33'29 2024-09-16T15:22:29.192062+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:16.788943+0000 43'29 91:95 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T22:00:15.112885+0000 2 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:16.781114+0000 41'29 91:109 [2,1,0] 2 [2,1,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:40:56.328702+0000 3 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 1 1 0 0 2105 0 0 18 12 18 active+recovery_wait+degraded 2024-09-16T15:24:16.785001+0000 85'30 91:107 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:16.783202+0000 39'29 91:93 [2,0,1] 2 [2,0,1] 2 39'29 2024-09-16T15:22:32.471614+0000 39'29 2024-09-16T15:22:32.471614+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:17.221479+0000 43'30 92:101 [0,2,1] 0 [0,2,1] 0 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 1 0 0 1755 0 0 17 12 17 active+recovery_wait+degraded 2024-09-16T15:24:17.299479+0000 33'29 92:89 [0,1,2] 0 [0,1,2] 0 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:24:17.180759+0000 33'28 92:119 [0,1,2] 0 [0,1,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:11:37.102392+0000 1 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:16.786041+0000 43'30 91:88 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.783014+0000 44'29 91:93 [2,0,1] 2 [2,0,1] 2 44'29 2024-09-16T15:22:31.091358+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:16.690375+0000 39'29 92:84 [1,0,2] 1 [1,0,2] 1 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:24:16.779750+0000 40'31 91:96 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.219 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 520 0 0 15 12 15 active+clean 2024-09-16T15:24:17.221460+0000 31'27 92:76 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.219 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:24:16.784154+0000 44'30 91:94 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.219 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1137 0 0 16 12 16 active+clean 2024-09-16T15:24:16.790676+0000 33'28 91:82 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.219 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:24:16.785338+0000 41'30 91:97 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.219 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 1 1 0 0 2988 0 0 19 12 19 active+recovering+degraded 2024-09-16T15:24:16.789686+0000 44'31 91:117 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:23.219 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:24:23.219 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:24:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:24:23.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:24:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:24:23.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:24:23.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:24:23.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:24:23.237 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:24:23.237 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:23.237 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:24:23.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:24:23.239 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:24:23.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:24:23.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:24:23.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-16T15:24:23.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:24:23.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:24:23.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:24:23.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:24:23.273 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:23.274+0000 7f8557f287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:23.276 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:23.278+0000 7f8557f287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:23.277 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:23.278+0000 7f8557f287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:23.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T15:24:23.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:24:23.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:24:23.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:24:23.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:24:23.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:24:23.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:24:23.666 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:24:23.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:24:23.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:24:23.849 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:23.850+0000 7f8557f287c0 -1 Falling back to public interface 2024-09-16T15:24:24.158 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 64 last_clean_interval [10,61) [v2:127.0.0.1:6818/4287159237,v1:127.0.0.1:6819/4287159237] [v2:127.0.0.1:6820/4287159237,v1:127.0.0.1:6821/4287159237] exists,up 0c65ed33-581c-493f-b80f-f44168583347 2024-09-16T15:24:24.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:24:24.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:24:24.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:24:24.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-09-16T15:24:24.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:24:24.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:24:24.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:24:24.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:24:24.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:24:24.160 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:24:24.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:24:24.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:24:24.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:24:24.598 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 64 last_clean_interval [10,61) [v2:127.0.0.1:6818/4287159237,v1:127.0.0.1:6819/4287159237] [v2:127.0.0.1:6820/4287159237,v1:127.0.0.1:6821/4287159237] exists,up 0c65ed33-581c-493f-b80f-f44168583347 2024-09-16T15:24:24.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:24:24.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:24:24.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:24:24.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-09-16T15:24:26.749 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:26.750+0000 7f8557f287c0 -1 osd.1 92 log_to_monitors true 2024-09-16T15:24:44.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-09-16T15:24:44.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T15:24:44.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T15:24:44.602 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:24:44.603 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:44.603 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:44.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.1.asok 2024-09-16T15:24:44.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-09-16T15:24:44.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T15:24:44.725 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:24:44.725 INFO:tasks.workunit.client.0.smithi039.stdout: "ops": [], 2024-09-16T15:24:44.725 INFO:tasks.workunit.client.0.smithi039.stdout: "num_ops": 0 2024-09-16T15:24:44.726 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:24:44.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-09-16T15:24:44.739 INFO:tasks.workunit.client.0.smithi039.stdout:allowing recovery 2024-09-16T15:24:44.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:24:45.157 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:24:45.157 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:28.718367+0000 39'29 97:109 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.157 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:28.749327+0000 44'29 97:113 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.157 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:24:28.717072+0000 40'31 97:118 [2,1,0] 2 [2,1,0] 2 40'31 2024-09-16T15:22:23.501248+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.157 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:28.743897+0000 43'29 97:116 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.157 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:28.748772+0000 35'29 97:96 [1,2,0] 1 [1,2,0] 1 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T23:22:09.306246+0000 3 0 2024-09-16T15:24:45.157 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 3 1 1 0 0 1755 0 0 17 12 17 active+recovery_wait+degraded 2024-09-16T15:24:28.743877+0000 41'29 97:98 [1,2,0] 1 [1,2,0] 1 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.157 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:28.744184+0000 43'30 97:119 [1,2,0] 1 [1,2,0] 1 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.157 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1138 0 0 16 12 16 active+clean 2024-09-16T15:24:28.720703+0000 35'28 97:120 [1,2,0] 1 [1,2,0] 1 35'28 2024-09-16T15:22:32.069055+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 5 0 0 0 0 2989 0 0 19 12 19 active+clean 2024-09-16T15:24:28.745947+0000 92'31 97:96 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-16T15:24:28.736631+0000 42'32 97:135 [2,0,1] 2 [2,0,1] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T19:11:52.379026+0000 6 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-16T15:24:28.745372+0000 36'28 97:98 [1,0,2] 1 [1,0,2] 1 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:57:39.055584+0000 1 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:28.745212+0000 44'30 97:110 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:24:28.736230+0000 40'28 97:124 [0,1,2] 0 [0,1,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T03:05:25.407168+0000 1 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:28.722086+0000 41'29 97:104 [1,0,2] 1 [1,0,2] 1 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-16T15:24:28.716953+0000 41'29 97:116 [2,0,1] 2 [2,0,1] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T20:05:28.203371+0000 0 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:28.743182+0000 33'29 97:120 [1,0,2] 1 [1,0,2] 1 33'29 2024-09-16T15:22:29.192062+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:28.749888+0000 43'29 97:106 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:34:49.238764+0000 2 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:28.718869+0000 41'29 97:124 [2,1,0] 2 [2,1,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T15:32:38.700390+0000 3 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 1 1 0 0 2105 0 0 18 12 18 active+recovery_wait+degraded 2024-09-16T15:24:28.746375+0000 85'30 97:117 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:28.737237+0000 39'29 97:108 [2,0,1] 2 [2,0,1] 2 39'29 2024-09-16T15:22:32.471614+0000 39'29 2024-09-16T15:22:32.471614+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:28.719314+0000 43'30 97:115 [0,2,1] 0 [0,2,1] 0 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 1 0 0 1755 0 0 17 12 17 active+recovering+degraded 2024-09-16T15:24:28.744948+0000 33'29 97:103 [0,1,2] 0 [0,1,2] 0 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.158 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:24:28.721111+0000 33'28 97:133 [0,1,2] 0 [0,1,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T21:06:51.975612+0000 1 0 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:28.744566+0000 43'30 97:99 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:28.738044+0000 44'29 97:108 [2,0,1] 2 [2,0,1] 2 44'29 2024-09-16T15:22:31.091358+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:28.746472+0000 39'29 97:94 [1,0,2] 1 [1,0,2] 1 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:24:28.739727+0000 40'31 97:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 520 0 0 15 12 15 active+clean 2024-09-16T15:24:28.721593+0000 31'27 97:90 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:24:28.680931+0000 44'30 97:109 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1137 0 0 16 12 16 active+clean 2024-09-16T15:24:28.748276+0000 33'28 97:93 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:24:28.744652+0000 41'30 97:108 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 1 1 0 0 2988 0 0 19 12 19 active+recovery_wait+degraded 2024-09-16T15:24:28.722472+0000 44'31 97:126 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:24:45.159 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:24:45.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-16T15:24:45.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-16T15:24:45.334 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:24:45.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-16T15:24:45.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-16T15:24:45.508 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:24:45.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-16T15:24:45.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-16T15:24:45.707 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:24:45.722 INFO:tasks.workunit.client.0.smithi039.stdout:reading divergent objects 2024-09-16T15:24:45.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-09-16T15:24:45.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-09-16T15:24:46.135 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:24:46.135 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:28.718367+0000 39'29 97:109 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.135 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:28.749327+0000 44'29 97:113 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.135 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:24:28.717072+0000 40'31 97:118 [2,1,0] 2 [2,1,0] 2 40'31 2024-09-16T15:22:23.501248+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.135 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:28.743897+0000 43'29 97:116 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.135 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:28.748772+0000 35'29 97:96 [1,2,0] 1 [1,2,0] 1 35'29 2024-09-16T15:22:10.125446+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T23:22:09.306246+0000 3 0 2024-09-16T15:24:46.135 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 3 1 1 0 0 1755 0 0 17 12 17 active+recovery_wait+degraded 2024-09-16T15:24:28.743877+0000 41'29 97:98 [1,2,0] 1 [1,2,0] 1 41'29 2024-09-16T15:22:17.647797+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:28.744184+0000 43'30 97:119 [1,2,0] 1 [1,2,0] 1 43'30 2024-09-16T15:22:15.055235+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1138 0 0 16 12 16 active+clean 2024-09-16T15:24:28.720703+0000 35'28 97:120 [1,2,0] 1 [1,2,0] 1 35'28 2024-09-16T15:22:32.069055+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 5 0 0 0 0 2989 0 0 19 12 19 active+clean 2024-09-16T15:24:28.745947+0000 92'31 97:96 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-16T15:24:28.736631+0000 42'32 97:135 [2,0,1] 2 [2,0,1] 2 42'32 2024-09-16T15:22:18.581350+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T19:11:52.379026+0000 6 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-16T15:24:28.745372+0000 36'28 97:98 [1,0,2] 1 [1,0,2] 1 31'27 2024-09-16T15:22:02.244252+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T16:57:39.055584+0000 1 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:28.745212+0000 44'30 97:110 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:24:28.736230+0000 40'28 97:124 [0,1,2] 0 [0,1,2] 0 40'28 2024-09-16T15:22:19.240655+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T03:05:25.407168+0000 1 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:28.722086+0000 41'29 97:104 [1,0,2] 1 [1,0,2] 1 41'29 2024-09-16T15:22:12.130300+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-16T15:24:28.716953+0000 41'29 97:116 [2,0,1] 2 [2,0,1] 2 31'27 2024-09-16T15:22:04.583723+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T20:05:28.203371+0000 0 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:28.743182+0000 33'29 97:120 [1,0,2] 1 [1,0,2] 1 33'29 2024-09-16T15:22:29.192062+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:28.749888+0000 43'29 97:106 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:00.096635+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-18T02:34:49.238764+0000 2 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-16T15:24:28.718869+0000 41'29 97:124 [2,1,0] 2 [2,1,0] 2 41'29 2024-09-16T15:22:19.647984+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T15:32:38.700390+0000 3 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 3 1 1 0 0 2105 0 0 18 12 18 active+recovery_wait+degraded 2024-09-16T15:24:28.746375+0000 85'30 97:117 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.136 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 3 0 0 0 0 1755 0 0 17 12 17 active+clean 2024-09-16T15:24:28.737237+0000 39'29 97:108 [2,0,1] 2 [2,0,1] 2 39'29 2024-09-16T15:22:32.471614+0000 39'29 2024-09-16T15:22:32.471614+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:28.719314+0000 43'30 97:115 [0,2,1] 0 [0,2,1] 0 43'30 2024-09-16T15:22:20.039422+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 1 0 0 1755 0 0 17 12 17 active+recovering+degraded 2024-09-16T15:24:28.744948+0000 33'29 97:103 [0,1,2] 0 [0,1,2] 0 33'29 2024-09-16T15:22:07.611155+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-16T15:24:28.721111+0000 33'28 97:133 [0,1,2] 0 [0,1,2] 0 33'28 2024-09-16T15:22:16.229813+0000 0'0 2024-09-16T15:21:37.776941+0000 0 1 periodic scrub scheduled @ 2024-09-17T21:06:51.975612+0000 1 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-16T15:24:28.744566+0000 43'30 97:99 [1,0,2] 1 [1,0,2] 1 33'28 2024-09-16T15:22:06.589927+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:28.738044+0000 44'29 97:108 [2,0,1] 2 [2,0,1] 2 44'29 2024-09-16T15:22:31.091358+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1754 0 0 17 12 17 active+clean 2024-09-16T15:24:28.746472+0000 39'29 97:94 [1,0,2] 1 [1,0,2] 1 39'29 2024-09-16T15:22:12.568944+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 5 0 0 0 0 2988 0 0 19 12 19 active+clean 2024-09-16T15:24:28.739727+0000 40'31 97:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 520 0 0 15 12 15 active+clean 2024-09-16T15:24:28.721593+0000 31'27 97:90 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:24:28.680931+0000 44'30 97:109 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1137 0 0 16 12 16 active+clean 2024-09-16T15:24:28.748276+0000 33'28 97:93 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2371 0 0 18 12 18 active+clean 2024-09-16T15:24:28.744652+0000 41'30 97:108 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 1 1 0 0 2988 0 0 19 12 19 active+recovery_wait+degraded 2024-09-16T15:24:28.722472+0000 44'31 97:126 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:21:37.776941+0000 0'0 2024-09-16T15:21:37.776941+0000 0 0 queued for scrub 0 0 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:24:46.137 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:24:46.138 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:24:46.153 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-09-16T15:24:46.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-09-16T15:24:46.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:24:46.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-09-16T15:24:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:24:46.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-09-16T15:24:46.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:24:46.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-09-16T15:24:46.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:24:46.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-09-16T15:24:46.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-09-16T15:24:46.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-16T15:24:46.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-16T15:24:46.340 INFO:tasks.workunit.client.0.smithi039.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-16T15:24:46.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-16T15:24:46.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-16T15:24:46.375 INFO:tasks.workunit.client.0.smithi039.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-16T15:24:46.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-09-16T15:24:46.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-16T15:24:46.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-16T15:24:46.401 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.043+0000 7fe655f77640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 51'30 (30'4) delete 1:70b3bf12:::existing_4:head by client.4637.0:1 2024-09-16T15:22:22.976865+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:24:46.401 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.043+0000 7fe655f77640 20 _merge_object_divergent_entries: keeping 51'30 (30'4) delete 1:70b3bf12:::existing_4:head by client.4637.0:1 2024-09-16T15:22:22.976865+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:24:46.401 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.043+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:24:46.401 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.043+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 30'4 first_divergent_update: 51'30 last_divergent_update: 51'30 2024-09-16T15:24:46.401 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.043+0000 7fe655f77640 10 _merge_object_divergent_entries: more recent entry found: 85'30 (30'4) modify 1:70b3bf12:::existing_4:head by client.4756.0:1 2024-09-16T15:23:28.646231+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0, already merged 2024-09-16T15:24:46.401 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.043+0000 7fe655f77640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 51'31 (0'0) modify 1:75ac5091:::newobject_2:head by client.4638.0:1 2024-09-16T15:22:22.977431+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-16T15:24:46.401 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.043+0000 7fe655f77640 20 _merge_object_divergent_entries: keeping 51'31 (0'0) modify 1:75ac5091:::newobject_2:head by client.4638.0:1 2024-09-16T15:22:22.977431+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-16T15:24:46.401 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.043+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:24:46.401 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.043+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 51'31 last_divergent_update: 51'31 2024-09-16T15:24:46.401 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.043+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.043+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 52'33 (30'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-09-16T15:22:22.979702+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 20 _merge_object_divergent_entries: keeping 52'33 (30'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-09-16T15:22:22.979702+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 30'1 first_divergent_update: 52'33 last_divergent_update: 52'33 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 52'33 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 52'33 (30'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-09-16T15:22:22.979702+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 30'1 <= info.log_tail 30'12 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 52'32 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4634.0:1 2024-09-16T15:22:22.975318+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 20 _merge_object_divergent_entries: keeping 52'32 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4634.0:1 2024-09-16T15:22:22.975318+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:24:46.402 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 52'32 last_divergent_update: 52'32 2024-09-16T15:24:46.403 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-16T15:24:46.403 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-16T15:24:46.403 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe655f77640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 51'30 (30'2) modify 1:97db2a0f:::existing_2:head by client.4636.0:1 2024-09-16T15:22:22.975416+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:24:46.403 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe655f77640 20 _merge_object_divergent_entries: keeping 51'30 (30'2) modify 1:97db2a0f:::existing_2:head by client.4636.0:1 2024-09-16T15:22:22.975416+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:24:46.403 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:24:46.403 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 30'2 first_divergent_update: 51'30 last_divergent_update: 51'30 2024-09-16T15:24:46.403 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-16T15:24:46.403 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-16T15:24:46.403 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 51'30 2024-09-16T15:24:46.412 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 51'30 (30'2) modify 1:97db2a0f:::existing_2:head by client.4636.0:1 2024-09-16T15:22:22.975416+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:24:46.412 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-16T15:24:46.412 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:16.155+0000 7fe655f77640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 30'2 <= info.log_tail 30'12 2024-09-16T15:24:46.412 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:17.095+0000 7fe656f79640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 51'30 (30'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-16T15:22:22.977223+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:24:46.412 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:17.095+0000 7fe656f79640 20 _merge_object_divergent_entries: keeping 51'30 (30'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-16T15:22:22.977223+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:24:46.412 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:17.095+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-16T15:24:46.412 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:17.095+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 30'3 first_divergent_update: 51'30 last_divergent_update: 51'30 2024-09-16T15:24:46.412 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:17.095+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-16T15:24:46.412 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:17.095+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-16T15:24:46.412 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:17.095+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 51'30 2024-09-16T15:24:46.413 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:17.095+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 51'30 (30'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-16T15:22:22.977223+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:24:46.413 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:17.095+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-16T15:24:46.413 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:24:17.095+0000 7fe656f79640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 30'3 <= info.log_tail 30'12 2024-09-16T15:24:46.413 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-16T15:24:16.043+0000 7fdc51fa0640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 51'30 (30'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-16T15:22:22.977223+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:24:46.413 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-16T15:24:16.043+0000 7fdc51fa0640 20 _merge_object_divergent_entries: keeping 51'30 (30'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-16T15:22:22.977223+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:24:46.413 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-16T15:24:16.043+0000 7fdc51fa0640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-16T15:24:46.413 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-16T15:24:16.043+0000 7fdc51fa0640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 30'3 first_divergent_update: 51'30 last_divergent_update: 51'30 2024-09-16T15:24:46.413 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-16T15:24:16.043+0000 7fdc51fa0640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-16T15:24:46.413 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-16T15:24:16.043+0000 7fdc51fa0640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-16T15:24:46.414 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-16T15:24:16.043+0000 7fdc51fa0640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 51'30 2024-09-16T15:24:46.414 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-16T15:24:16.043+0000 7fdc51fa0640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 51'30 (30'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-16T15:22:22.977223+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2024-09-16T15:24:46.414 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-16T15:24:16.043+0000 7fdc51fa0640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-16T15:24:46.414 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-16T15:24:16.043+0000 7fdc51fa0640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 30'3 <= info.log_tail 30'12 2024-09-16T15:24:46.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-16T15:24:46.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-16T15:24:46.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-16T15:24:46.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-16T15:24:46.455 INFO:tasks.workunit.client.0.smithi039.stdout:success 2024-09-16T15:24:46.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-09-16T15:24:46.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.27113.out 2024-09-16T15:24:46.456 INFO:tasks.workunit.client.0.smithi039.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27113.out': No such file or directory 2024-09-16T15:24:46.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-09-16T15:24:46.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T15:24:46.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T15:24:47.111 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T15:24:47.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-09-16T15:24:47.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:24:47.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:24:47.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:24:47.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:24:47.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:24:47.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:24:47.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-16T15:24:47.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-16T15:24:47.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:24:47.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-16T15:24:47.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:24:47.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:24:47.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:24:47.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:24:47.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:24:47.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:24:47.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:24:47.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:24:47.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:24:47.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:24:47.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:24:47.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:24:47.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:24:47.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:24:47.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:24:47.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:24:47.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:24:47.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:24:47.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-16T15:24:47.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:24:47.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:47.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:47.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27113 2024-09-16T15:24:47.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:24:47.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:24:47.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-16T15:24:47.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-16T15:24:47.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-16T15:24:47.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-16T15:24:47.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-16T15:24:47.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:24:47.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-16T15:24:47.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:24:47.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:24:47.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:24:47.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:24:47.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:24:47.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:24:47.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:24:47.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:24:47.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:24:47.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:24:47.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:24:47.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:24:47.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:24:47.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:24:47.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:24:47.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:24:47.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:24:47.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:24:47.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-16T15:24:47.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:24:47.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:47.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:47.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27113 2024-09-16T15:24:47.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:24:47.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:24:47.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-16T15:24:47.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:24:47.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:47.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:47.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27113 2024-09-16T15:24:47.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:24:47.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T15:24:47.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:24:47.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-16T15:24:47.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-09-16T15:24:47.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-09-16T15:24:47.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-09-16T15:24:47.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-09-16T15:24:47.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-09-16T15:24:47.513 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-09-16T15:24:47.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-09-16T15:24:47.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-09-16T15:24:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:24:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:24:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-09-16T15:24:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-16T15:24:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:24:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:24:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:24:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-16T15:24:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-16T15:24:47.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:24:47.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:24:47.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:24:47.904 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:24:47.904 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:47.904 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:47.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:24:47.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:24:48.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:24:48.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:24:48.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:24:48.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:24:48.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:24:48.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:24:48.136 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:24:48.136 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:24:48.136 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:24:48.136 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:24:48.137 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:48.137 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:48.137 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-mon.a.asok 2024-09-16T15:24:48.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:24:48.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27113/ceph-mon.a.asok config get fsid 2024-09-16T15:24:48.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:24:48.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:24:48.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:24:48.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:24:48.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:24:48.257 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:24:48.257 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:24:48.257 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:24:48.257 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:24:48.257 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:48.258 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:48.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-mon.a.asok 2024-09-16T15:24:48.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:24:48.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27113/ceph-mon.a.asok config get mon_host 2024-09-16T15:24:48.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-09-16T15:24:48.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-16T15:24:48.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:24:48.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:24:48.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:24:48.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-16T15:24:48.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:24:49.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:24:49.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:24:49.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:24:49.301 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:24:49.301 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:49.301 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:49.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:24:49.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:24:49.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:24:49.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-16T15:24:49.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-09-16T15:24:49.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:24:49.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:24:49.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:24:49.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:24:49.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:24:49.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:24:49.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:24:49.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:24:49.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-16T15:24:49.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:24:49.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb303f60-3b45-4945-a069-9f2f1ab9cc77 2024-09-16T15:24:49.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 eb303f60-3b45-4945-a069-9f2f1ab9cc77' 2024-09-16T15:24:49.372 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 eb303f60-3b45-4945-a069-9f2f1ab9cc77 2024-09-16T15:24:49.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:24:49.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBTehm/nG+FxAA0MwQpZJJROcvn7yiiARixA== 2024-09-16T15:24:49.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBTehm/nG+FxAA0MwQpZJJROcvn7yiiARixA=="}' 2024-09-16T15:24:49.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb303f60-3b45-4945-a069-9f2f1ab9cc77 -i td/divergent-priors/0/new.json 2024-09-16T15:24:49.728 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:24:49.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-16T15:24:49.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBTehm/nG+FxAA0MwQpZJJROcvn7yiiARixA== --osd-uuid eb303f60-3b45-4945-a069-9f2f1ab9cc77 2024-09-16T15:24:49.788 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:49.790+0000 7fd1379647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:49.790 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:49.790+0000 7fd1379647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:49.792 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:49.794+0000 7fd1379647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:49.792 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:49.794+0000 7fd1379647c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:24:49.792 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:49.794+0000 7fd1379647c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:24:49.792 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:49.794+0000 7fd1379647c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-16T15:24:49.831 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:49.830+0000 7fd1379647c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-16T15:24:56.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-16T15:24:56.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:24:56.155 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:24:56.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:24:56.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:24:56.725 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:24:56.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:24:56.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:24:56.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:24:56.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:24:56.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:24:56.767 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:56.766+0000 7fbf914ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:56.782 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:56.782+0000 7fbf914ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:56.783 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:56.786+0000 7fbf914ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:24:57.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:24:57.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:24:57.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:24:57.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:24:57.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:24:57.142 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:24:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:24:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:24:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:24:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:24:57.390 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:57.390+0000 7fbf914ec7c0 -1 Falling back to public interface 2024-09-16T15:24:57.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:24:58.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:24:58.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:24:58.593 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:24:58.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:24:58.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:24:58.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:24:59.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:24:59.905 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:24:59.906+0000 7fbf914ec7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:25:00.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:00.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:00.030 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:25:00.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:25:00.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:00.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:00.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:01.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:01.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:01.504 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:25:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:25:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:01.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:02.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:02.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:02.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:25:02.958 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:25:02.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:02.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:03.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:04.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:04.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:04.423 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:25:04.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:25:04.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:04.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:04.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:05.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:05.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:05.948 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:25:05.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:25:05.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:05.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:06.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:07.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:07.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:07.395 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:25:07.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:25:07.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:07.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:07.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:08.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:08.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:08.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:25:08.928 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:25:08.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:08.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:09.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:10.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:10.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:10.379 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:25:10.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:25:10.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:10.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:10.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:11.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:11.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:11.940 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:25:11.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:25:11.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:11.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:12.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:13.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:13.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:13.406 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:25:13.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:25:13.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:13.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:14.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:15.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:15.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:15.329 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:25:15.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:25:15.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:15.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:16.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:17.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:17.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:17.166 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:25:17.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:25:17.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:17.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:17.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:18.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:18.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:18.904 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:25:18.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:25:18.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:18.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:20.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:21.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:21.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:21.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:25:21.058 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:25:21.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:21.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:21.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:22.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:22.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:22.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:25:22.952 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:25:22.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:22.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:23.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:24.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:24.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:24.599 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:25:24.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:25:24.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:24.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:25:25.078 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/251805189,v1:127.0.0.1:6803/251805189] [v2:127.0.0.1:6804/251805189,v1:127.0.0.1:6805/251805189] exists,up eb303f60-3b45-4945-a069-9f2f1ab9cc77 2024-09-16T15:25:25.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:25:25.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:25:25.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:25:25.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-16T15:25:25.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-09-16T15:25:25.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:25:25.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:25:25.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:25:25.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:25:25.082 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:25:25.083 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:25:25.083 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:25:25.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:25:25.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:25:25.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:25:25.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:25:25.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:25:25.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:25:25.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:25:25.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:25:25.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:25:25.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:25:25.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:25:25.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:25:25.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:25:25.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:25:25.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:25:25.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=409fbc6b-a686-4358-b419-40a385379580 2024-09-16T15:25:25.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 409fbc6b-a686-4358-b419-40a385379580' 2024-09-16T15:25:25.086 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 409fbc6b-a686-4358-b419-40a385379580 2024-09-16T15:25:25.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:25:25.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDlTehmcY8HBxAAgQG37d+UiX/203PD8lcf4Q== 2024-09-16T15:25:25.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDlTehmcY8HBxAAgQG37d+UiX/203PD8lcf4Q=="}' 2024-09-16T15:25:25.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 409fbc6b-a686-4358-b419-40a385379580 -i td/divergent-priors/1/new.json 2024-09-16T15:25:25.683 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:25:25.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-16T15:25:25.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlTehmcY8HBxAAgQG37d+UiX/203PD8lcf4Q== --osd-uuid 409fbc6b-a686-4358-b419-40a385379580 2024-09-16T15:25:25.749 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:25.749+0000 7fa835bb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:25:25.772 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:25.773+0000 7fa835bb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:25:25.773 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:25.773+0000 7fa835bb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:25:25.774 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:25.773+0000 7fa835bb67c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:25:25.774 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:25.773+0000 7fa835bb67c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:25:25.774 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:25.773+0000 7fa835bb67c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-16T15:25:25.864 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:25.865+0000 7fa835bb67c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-16T15:25:31.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-16T15:25:31.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:25:31.984 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:25:31.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:25:31.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:25:32.600 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:25:32.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:25:32.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:25:32.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:25:32.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:25:32.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:25:32.642 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:32.641+0000 7f13be1c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:25:32.645 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:32.645+0000 7f13be1c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:25:32.646 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:32.645+0000 7f13be1c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:25:33.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:25:33.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:25:33.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:25:33.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:25:33.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:25:33.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:33.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:25:33.014 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:25:33.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:33.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:33.241 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:33.241+0000 7f13be1c17c0 -1 Falling back to public interface 2024-09-16T15:25:33.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:34.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:34.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:34.495 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:25:34.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:25:34.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:34.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:34.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:35.906 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:25:35.905+0000 7f13be1c17c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:25:35.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:35.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:35.991 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:25:35.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:25:35.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:35.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:36.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:37.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:37.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:37.483 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:25:37.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:25:37.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:37.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:37.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:38.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:38.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:38.930 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:25:38.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:25:38.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:38.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:39.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:40.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:40.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:40.370 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:25:40.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:25:40.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:40.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:40.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:41.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:41.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:41.815 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:25:41.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:25:41.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:41.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:42.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:43.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:43.352 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:25:43.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:43.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:25:43.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:43.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:43.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:44.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:44.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:44.812 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:25:44.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:25:44.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:44.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:45.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:46.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:46.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:46.305 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:25:46.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:25:46.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:46.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:46.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:47.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:47.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:47.894 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:25:47.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:25:47.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:47.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:48.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:49.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:49.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:49.340 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:25:49.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:25:49.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:49.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:49.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:50.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:50.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:50.808 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:25:50.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:25:50.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:50.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:51.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:52.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:52.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:52.408 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:25:52.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:25:52.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:52.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:52.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:53.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:53.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:53.949 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:25:53.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:25:53.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:53.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:54.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:55.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:55.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:55.658 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:25:55.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:25:55.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:55.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:56.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:57.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:57.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:57.473 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:25:57.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:25:57.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:57.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:58.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:25:59.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:25:59.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:25:59.531 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:25:59.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:25:59.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:25:59.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:25:59.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:01.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:01.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:01.001 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T15:26:01.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T15:26:01.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:01.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:26:01.501 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1133638368,v1:127.0.0.1:6811/1133638368] [v2:127.0.0.1:6812/1133638368,v1:127.0.0.1:6813/1133638368] exists,up 409fbc6b-a686-4358-b419-40a385379580 2024-09-16T15:26:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:26:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:26:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:26:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-16T15:26:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-09-16T15:26:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-16T15:26:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:26:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T15:26:01.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:26:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:26:01.506 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:26:01.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:26:01.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:26:01.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-16T15:26:01.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:26:01.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=42d6a639-404a-47f9-b44b-75f4fe86cfa5 2024-09-16T15:26:01.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 42d6a639-404a-47f9-b44b-75f4fe86cfa5' 2024-09-16T15:26:01.509 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 42d6a639-404a-47f9-b44b-75f4fe86cfa5 2024-09-16T15:26:01.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:26:01.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJTuhmLasRIBAACOAK7p+UpvCmj2W2UDIgwg== 2024-09-16T15:26:01.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJTuhmLasRIBAACOAK7p+UpvCmj2W2UDIgwg=="}' 2024-09-16T15:26:01.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 42d6a639-404a-47f9-b44b-75f4fe86cfa5 -i td/divergent-priors/2/new.json 2024-09-16T15:26:02.085 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:26:02.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-16T15:26:02.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJTuhmLasRIBAACOAK7p+UpvCmj2W2UDIgwg== --osd-uuid 42d6a639-404a-47f9-b44b-75f4fe86cfa5 2024-09-16T15:26:02.156 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:02.157+0000 7fe9047377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:26:02.165 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:02.165+0000 7fe9047377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:26:02.167 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:02.169+0000 7fe9047377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:26:02.167 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:02.169+0000 7fe9047377c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:26:02.167 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:02.169+0000 7fe9047377c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:26:02.167 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:02.169+0000 7fe9047377c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-16T15:26:02.291 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:02.293+0000 7fe9047377c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-16T15:26:08.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-16T15:26:08.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:26:08.406 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T15:26:08.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T15:26:08.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:26:08.971 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:26:08.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T15:26:08.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:26:08.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:26:08.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:26:08.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:26:09.014 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:09.012+0000 7f578e7db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:26:09.041 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:09.040+0000 7f578e7db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:26:09.042 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:09.040+0000 7f578e7db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:26:09.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T15:26:09.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:26:09.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:26:09.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:26:09.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:26:09.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:09.424 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:26:09.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:26:09.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:09.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:09.600 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:09.600+0000 7f578e7db7c0 -1 Falling back to public interface 2024-09-16T15:26:09.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:10.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:10.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:10.882 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:26:10.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:26:10.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:10.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:11.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:12.259 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:26:12.260+0000 7f578e7db7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:26:12.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:12.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:12.360 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:26:12.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:26:12.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:12.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:12.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:13.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:13.893 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:26:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:26:13.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:13.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:14.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:15.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:15.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:15.331 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:26:15.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:26:15.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:15.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:15.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:16.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:16.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:16.816 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:26:16.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:26:16.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:16.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:17.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:18.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:18.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:18.351 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:26:18.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:26:18.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:18.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:18.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:19.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:19.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:19.855 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:26:19.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:26:19.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:19.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:20.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:21.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:21.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:21.491 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:26:21.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:26:21.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:21.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:21.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:22.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:22.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:22.986 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:26:22.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:26:22.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:22.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:23.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:24.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:24.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:24.558 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:26:24.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:26:24.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:24.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:25.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:26.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:26.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:26.193 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:26:26.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:26:26.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:26.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:27.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:28.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:28.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:28.531 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:26:28.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:26:28.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:28.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:29.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:30.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:30.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:30.441 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:26:30.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:26:30.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:30.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:31.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:32.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:32.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:32.782 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:26:32.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:26:32.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:32.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:33.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:34.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:34.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:34.777 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:26:34.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:26:34.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:34.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:35.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:26:36.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:26:36.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:26:36.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:26:36.990 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:26:36.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:26:36.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:26:37.484 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3439842056,v1:127.0.0.1:6819/3439842056] [v2:127.0.0.1:6820/3439842056,v1:127.0.0.1:6821/3439842056] exists,up 42d6a639-404a-47f9-b44b-75f4fe86cfa5 2024-09-16T15:26:37.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:26:37.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:26:37.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:26:37.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-09-16T15:26:38.217 INFO:tasks.workunit.client.0.smithi039.stderr:noout is set 2024-09-16T15:26:38.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-09-16T15:26:39.197 INFO:tasks.workunit.client.0.smithi039.stderr:noin is set 2024-09-16T15:26:39.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-09-16T15:26:40.017 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T15:26:40.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-09-16T15:26:40.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2390: create_ec_pool: local pool_name=test 2024-09-16T15:26:40.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2391: create_ec_pool: shift 2024-09-16T15:26:40.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2392: create_ec_pool: local allow_overwrites=true 2024-09-16T15:26:40.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: create_ec_pool: shift 2024-09-16T15:26:40.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-16T15:26:40.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-09-16T15:26:40.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-16T15:26:41.555 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T15:26:41.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:26:42.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: create_ec_pool: '[' true = true ']' 2024-09-16T15:26:42.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-09-16T15:26:43.390 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 allow_ec_overwrites to true 2024-09-16T15:26:43.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: create_ec_pool: wait_for_clean 2024-09-16T15:26:43.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:26:43.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:26:43.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:26:43.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:26:43.415 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:26:43.415 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:26:43.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:26:43.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:26:43.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:26:43.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:26:43.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:26:43.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:26:43.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:26:43.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:26:43.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:26:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:26:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:26:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:26:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:26:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:26:44.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:26:44.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836500 2024-09-16T15:26:44.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836500 2024-09-16T15:26:44.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500' 2024-09-16T15:26:44.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:26:44.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:26:44.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672975 2024-09-16T15:26:44.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672975 2024-09-16T15:26:44.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500 1-42949672975' 2024-09-16T15:26:44.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:26:44.398 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:26:44.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509449 2024-09-16T15:26:44.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509449 2024-09-16T15:26:44.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500 1-42949672975 2-64424509449' 2024-09-16T15:26:44.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:26:44.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836500 2024-09-16T15:26:44.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:26:44.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:26:44.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836500 2024-09-16T15:26:44.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:26:44.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836500 2024-09-16T15:26:44.588 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836500 2024-09-16T15:26:44.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-16T15:26:44.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:26:45.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-16T15:26:45.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:26:46.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:26:46.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:26:46.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836501 -lt 21474836500 2024-09-16T15:26:46.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:26:46.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672975 2024-09-16T15:26:46.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:26:46.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:26:46.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672975 2024-09-16T15:26:46.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:26:46.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672975 2024-09-16T15:26:46.481 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672975 2024-09-16T15:26:46.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-16T15:26:46.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:26:46.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-16T15:26:46.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:26:46.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509449 2024-09-16T15:26:46.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:26:46.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:26:46.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509449 2024-09-16T15:26:46.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:26:46.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509449 2024-09-16T15:26:46.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-16T15:26:46.922 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509449 2024-09-16T15:26:46.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:26:47.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-16T15:26:47.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:26:47.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:26:47.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:26:47.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:26:47.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:26:47.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:26:47.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:26:47.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:26:47.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:26:47.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:26:47.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:26:48.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T15:26:48.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:26:48.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:26:48.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:26:48.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T15:26:48.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:26:48.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:26:48.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: create_ec_pool: return 0 2024-09-16T15:26:48.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-09-16T15:26:48.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:26:48.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:26:49.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:26:49.296 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:26:49.296 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:26:49.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:26:49.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:26:49.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:26:49.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836504 2024-09-16T15:26:49.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836504 2024-09-16T15:26:49.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504' 2024-09-16T15:26:49.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:26:49.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:26:49.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672978 2024-09-16T15:26:49.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672978 2024-09-16T15:26:49.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504 1-42949672978' 2024-09-16T15:26:49.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:26:49.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:26:49.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509452 2024-09-16T15:26:49.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509452 2024-09-16T15:26:49.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504 1-42949672978 2-64424509452' 2024-09-16T15:26:49.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:26:49.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836504 2024-09-16T15:26:49.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:26:49.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:26:49.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836504 2024-09-16T15:26:49.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:26:49.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836504 2024-09-16T15:26:49.814 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836504 2024-09-16T15:26:49.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-16T15:26:49.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:26:50.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836501 -lt 21474836504 2024-09-16T15:26:50.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:26:51.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:26:51.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:26:51.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-16T15:26:51.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:26:51.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672978 2024-09-16T15:26:51.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:26:51.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:26:51.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672978 2024-09-16T15:26:51.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:26:51.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672978 2024-09-16T15:26:51.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-16T15:26:51.716 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672978 2024-09-16T15:26:51.717 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:26:52.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-16T15:26:52.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:26:52.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509452 2024-09-16T15:26:52.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:26:52.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:26:52.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509452 2024-09-16T15:26:52.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:26:52.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509452 2024-09-16T15:26:52.146 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509452 2024-09-16T15:26:52.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-16T15:26:52.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:26:52.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-16T15:26:52.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-09-16T15:26:52.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:26:52.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:26:52.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:26:52.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:26:52.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:26:52.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:26:52.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:26:52.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:26:52.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:26:52.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:26:52.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:26:52.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:26:52.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:26:52.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:26:52.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:26:53.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:26:53.143 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:26:53.143 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:26:53.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:26:53.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:26:53.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:26:53.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836506 2024-09-16T15:26:53.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836506 2024-09-16T15:26:53.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836506' 2024-09-16T15:26:53.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:26:53.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:26:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672981 2024-09-16T15:26:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672981 2024-09-16T15:26:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836506 1-42949672981' 2024-09-16T15:26:53.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:26:53.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:26:53.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509455 2024-09-16T15:26:53.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509455 2024-09-16T15:26:53.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836506 1-42949672981 2-64424509455' 2024-09-16T15:26:53.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:26:53.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836506 2024-09-16T15:26:53.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:26:53.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:26:53.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836506 2024-09-16T15:26:53.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:26:53.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836506 2024-09-16T15:26:53.641 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836506 2024-09-16T15:26:53.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-16T15:26:53.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:26:54.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836504 -lt 21474836506 2024-09-16T15:26:54.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:26:55.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:26:55.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:26:55.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-16T15:26:55.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:26:55.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:26:55.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672981 2024-09-16T15:26:55.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:26:55.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672981 2024-09-16T15:26:55.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:26:55.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672981 2024-09-16T15:26:55.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-16T15:26:55.479 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672981 2024-09-16T15:26:55.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:26:55.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-16T15:26:55.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:26:55.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509455 2024-09-16T15:26:55.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:26:55.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:26:55.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509455 2024-09-16T15:26:55.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:26:55.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509455 2024-09-16T15:26:55.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-16T15:26:55.903 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509455 2024-09-16T15:26:55.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:26:56.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-16T15:26:56.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:26:56.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:26:56.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:26:56.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:26:56.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:26:56.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:26:56.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:26:56.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:26:56.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:26:56.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:26:56.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:26:57.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T15:26:57.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:26:57.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:26:57.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:26:57.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T15:26:57.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:26:57.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:26:57.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-09-16T15:26:57.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-09-16T15:26:58.334 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:26:58.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-09-16T15:26:58.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-09-16T15:26:58.352 INFO:tasks.workunit.client.0.smithi039.stdout:primary and soon to be divergent is 1 2024-09-16T15:26:58.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:26:58.756 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:26:58.756 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T15:26:42.413941+0000 0'0 25:11 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:13:06.624705+0000 0 0 2024-09-16T15:26:58.756 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:26:58.756 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:26:58.756 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:26:58.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-09-16T15:26:58.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-16T15:26:58.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-09-16T15:26:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-09-16T15:26:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-16T15:26:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-09-16T15:26:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-09-16T15:26:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-16T15:26:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-09-16T15:26:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-09-16T15:26:58.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-09-16T15:26:58.774 INFO:tasks.workunit.client.0.smithi039.stdout:writing initial objects 2024-09-16T15:26:58.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-09-16T15:26:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:26:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-09-16T15:26:58.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:26:58.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-09-16T15:26:59.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:26:59.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-09-16T15:26:59.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:26:59.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-09-16T15:26:59.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:26:59.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-09-16T15:26:59.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:26:59.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-09-16T15:27:00.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-09-16T15:27:00.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:00.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-09-16T15:27:00.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:00.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-09-16T15:27:00.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:00.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-09-16T15:27:01.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:01.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-09-16T15:27:01.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:01.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-09-16T15:27:01.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:01.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-09-16T15:27:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-09-16T15:27:02.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:02.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-09-16T15:27:02.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:02.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-09-16T15:27:02.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:02.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-09-16T15:27:02.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:02.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-09-16T15:27:02.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:02.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-09-16T15:27:03.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:03.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-09-16T15:27:03.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:03.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-09-16T15:27:03.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:03.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-09-16T15:27:03.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:03.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-09-16T15:27:04.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:04.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-09-16T15:27:04.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:04.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-09-16T15:27:04.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:04.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-09-16T15:27:04.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:04.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-09-16T15:27:05.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:05.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-09-16T15:27:05.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:05.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-09-16T15:27:05.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:05.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-09-16T15:27:05.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:05.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-09-16T15:27:05.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:05.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-09-16T15:27:06.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:06.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-09-16T15:27:06.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:06.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-09-16T15:27:06.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:06.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-09-16T15:27:06.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:06.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-09-16T15:27:07.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:07.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-09-16T15:27:07.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:07.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-09-16T15:27:07.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:07.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-09-16T15:27:07.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:07.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-09-16T15:27:08.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:08.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-09-16T15:27:08.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:08.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-09-16T15:27:08.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:08.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-09-16T15:27:08.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:08.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-09-16T15:27:08.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:08.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-09-16T15:27:09.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:09.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-09-16T15:27:09.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:09.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-09-16T15:27:09.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:09.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-09-16T15:27:09.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:09.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-09-16T15:27:10.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:10.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-09-16T15:27:10.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:10.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-09-16T15:27:10.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:10.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-09-16T15:27:10.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:10.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-09-16T15:27:11.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:11.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-09-16T15:27:11.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:11.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-09-16T15:27:11.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:11.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-09-16T15:27:11.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:11.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-09-16T15:27:12.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:12.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-09-16T15:27:12.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:12.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-09-16T15:27:12.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:12.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-09-16T15:27:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-09-16T15:27:13.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:13.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-09-16T15:27:13.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:13.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-09-16T15:27:13.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:13.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-09-16T15:27:13.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:13.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-09-16T15:27:13.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:13.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-09-16T15:27:14.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:14.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-09-16T15:27:14.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:14.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-09-16T15:27:14.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:14.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-09-16T15:27:14.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:14.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-09-16T15:27:15.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:15.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-09-16T15:27:15.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:15.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-09-16T15:27:15.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:15.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-09-16T15:27:15.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:15.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-09-16T15:27:16.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:16.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-09-16T15:27:16.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:16.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-09-16T15:27:16.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:16.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-09-16T15:27:16.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:16.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-09-16T15:27:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:17.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-09-16T15:27:17.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:17.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-09-16T15:27:17.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:17.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-09-16T15:27:17.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:17.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-09-16T15:27:18.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:18.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-09-16T15:27:18.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:18.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-09-16T15:27:18.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:18.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-09-16T15:27:18.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:18.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-09-16T15:27:19.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:19.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-09-16T15:27:19.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:19.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-09-16T15:27:19.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:19.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-09-16T15:27:19.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:19.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-09-16T15:27:19.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:19.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-09-16T15:27:20.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:20.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-09-16T15:27:20.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:20.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-09-16T15:27:20.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:20.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-09-16T15:27:20.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:20.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-09-16T15:27:21.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:21.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-09-16T15:27:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-09-16T15:27:21.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:21.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-09-16T15:27:21.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:21.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-16T15:27:22.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-16T15:27:22.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-16T15:27:22.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T15:27:22.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-09-16T15:27:22.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:27:22.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:27:22.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:27:22.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T15:27:22.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:27:22.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:27:22.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:27:22.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:27:22.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:27:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T15:27:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:27:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:27:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:27:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:27:22.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:27:22.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:27:22.870 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:27:22.870 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:27:22.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:27:22.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:27:22.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:27:23.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836514 2024-09-16T15:27:23.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836514 2024-09-16T15:27:23.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836514' 2024-09-16T15:27:23.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:27:23.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:27:23.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672989 2024-09-16T15:27:23.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672989 2024-09-16T15:27:23.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836514 1-42949672989' 2024-09-16T15:27:23.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:27:23.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:27:23.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509463 2024-09-16T15:27:23.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509463 2024-09-16T15:27:23.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836514 1-42949672989 2-64424509463' 2024-09-16T15:27:23.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:27:23.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836514 2024-09-16T15:27:23.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:27:23.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:27:23.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836514 2024-09-16T15:27:23.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:27:23.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836514 2024-09-16T15:27:23.367 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836514 2024-09-16T15:27:23.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-09-16T15:27:23.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:27:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836512 -lt 21474836514 2024-09-16T15:27:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:27:24.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:27:24.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:27:25.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836514 -lt 21474836514 2024-09-16T15:27:25.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:27:25.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672989 2024-09-16T15:27:25.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:27:25.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:27:25.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672989 2024-09-16T15:27:25.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:27:25.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672989 2024-09-16T15:27:25.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-16T15:27:25.220 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672989 2024-09-16T15:27:25.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:27:25.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672989 -lt 42949672989 2024-09-16T15:27:25.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:27:25.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509463 2024-09-16T15:27:25.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:27:25.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:27:25.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509463 2024-09-16T15:27:25.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:27:25.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509463 2024-09-16T15:27:25.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-16T15:27:25.654 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509463 2024-09-16T15:27:25.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:27:26.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509463 -lt 64424509463 2024-09-16T15:27:26.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:27:26.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:27:26.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:27:26.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:27:26.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:27:26.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:27:26.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:27:26.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:27:26.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:27:26.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:27:26.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:27:27.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T15:27:27.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:27:27.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:27:27.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:27:27.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T15:27:27.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:27:27.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:27:27.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-09-16T15:27:27.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-16T15:27:27.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-16T15:27:27.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-16T15:27:27.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T15:27:28.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-09-16T15:27:28.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-09-16T15:27:28.075 INFO:tasks.workunit.client.0.smithi039.stdout:blackholing osds 0 2 2024-09-16T15:27:28.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:27:28.475 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:27:28.475 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-16T15:26:42.413941+0000 28'100 28:111 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:13:06.624705+0000 0 0 2024-09-16T15:27:28.475 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:27:28.475 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:27:28.475 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:27:28.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:27:28.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-09-16T15:27:28.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T15:27:28.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T15:27:28.494 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:27:28.494 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:27:28.494 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:27:28.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.0.asok 2024-09-16T15:27:28.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-16T15:27:28.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-16T15:27:28.607 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:27:28.607 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-16T15:27:28.607 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:27:28.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:27:28.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-09-16T15:27:28.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T15:27:28.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T15:27:28.622 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:27:28.622 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:27:28.622 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:27:28.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.2.asok 2024-09-16T15:27:28.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-16T15:27:28.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-16T15:27:28.731 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:27:28.731 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-16T15:27:28.731 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:27:28.743 INFO:tasks.workunit.client.0.smithi039.stdout:writing divergent object 2024-09-16T15:27:28.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-09-16T15:27:28.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-09-16T15:27:28.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-09-16T15:27:29.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-09-16T15:27:29.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-16T15:27:30.057 INFO:tasks.workunit.client.0.smithi039.stdout:created pool test snap snap1 2024-09-16T15:27:30.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-09-16T15:27:30.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-09-16T15:27:30.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-16T15:27:40.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-09-16T15:27:40.068 INFO:tasks.workunit.client.0.smithi039.stderr:rados: no process found 2024-09-16T15:27:40.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-09-16T15:27:40.068 INFO:tasks.workunit.client.0.smithi039.stdout:killing all the osds 2024-09-16T15:27:40.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:27:40.536 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:27:40.536 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 active+clean 2024-09-16T15:26:42.413941+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:13:06.624705+0000 0 0 2024-09-16T15:27:40.536 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:27:40.536 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:27:40.537 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:27:40.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-09-16T15:27:40.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:27:40.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:27:40.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:27:40.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:27:40.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:27:40.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:27:40.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-16T15:27:40.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-09-16T15:27:41.229 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T15:27:41.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-16T15:27:41.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-09-16T15:27:41.828 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-16T15:27:41.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-16T15:27:41.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-09-16T15:27:42.433 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-16T15:27:42.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:27:42.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-09-16T15:27:43.030 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T15:27:43.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:27:43.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-09-16T15:27:43.646 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already out. 2024-09-16T15:27:43.662 INFO:tasks.workunit.client.0.smithi039.stdout:bringing up non_divergent 0 2 2024-09-16T15:27:43.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-09-16T15:27:43.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:27:44.097 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:27:44.097 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 stale+active+clean 2024-09-16T15:26:42.413941+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:13:06.624705+0000 0 0 2024-09-16T15:27:44.097 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:27:44.097 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:27:44.097 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:27:44.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:27:44.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-16T15:27:44.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:27:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:27:44.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:27:44.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:27:44.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:27:44.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:27:44.118 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:27:44.118 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:27:44.118 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:27:44.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:27:44.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:27:44.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:27:44.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:27:44.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:27:44.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:27:44.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:27:44.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:27:44.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:27:44.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:27:44.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:27:44.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-16T15:27:44.119 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:27:44.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:27:44.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:27:44.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-16T15:27:44.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:27:44.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:27:44.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:27:44.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:27:44.167 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:27:44.170+0000 7f7219de97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:27:44.192 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:27:44.194+0000 7f7219de97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:27:44.194 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:27:44.194+0000 7f7219de97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:27:44.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:27:44.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:27:44.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:27:44.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:27:44.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:27:44.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:27:44.564 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:27:44.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:27:44.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:27:44.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:27:44.744 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:27:44.746+0000 7f7219de97c0 -1 Falling back to public interface 2024-09-16T15:27:45.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:27:46.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:27:46.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:27:46.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:27:46.051 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:27:46.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:27:46.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:27:46.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:27:47.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:27:47.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:27:47.604 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:27:47.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:27:47.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:27:47.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:27:47.710 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:27:47.710+0000 7f7219de97c0 -1 osd.0 28 log_to_monitors true 2024-09-16T15:27:48.129 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/3891233128,v1:127.0.0.1:6803/3891233128] [v2:127.0.0.1:6804/3891233128,v1:127.0.0.1:6805/3891233128] exists,up eb303f60-3b45-4945-a069-9f2f1ab9cc77 2024-09-16T15:27:48.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:27:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:27:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:27:48.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:27:48.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:27:48.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:27:48.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:27:48.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:27:48.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:27:48.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:27:48.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-16T15:27:48.135 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:27:48.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T15:27:48.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:27:48.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-16T15:27:48.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T15:27:48.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:27:48.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:27:48.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:27:48.176 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:27:48.178+0000 7f9a954367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:27:48.278 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:27:48.278+0000 7f9a954367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:27:48.280 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:27:48.282+0000 7f9a954367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:27:48.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T15:27:48.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:27:48.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:27:48.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:27:48.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:27:48.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:27:48.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:27:48.614 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:27:48.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:27:48.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:27:48.814 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:27:48.814+0000 7f9a954367c0 -1 Falling back to public interface 2024-09-16T15:27:49.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:27:50.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:27:50.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:27:50.049 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:27:50.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:27:50.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:27:50.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:27:50.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:27:51.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:27:51.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:27:51.505 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:27:51.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:27:51.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:27:51.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:27:51.796 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:27:51.798+0000 7f9a954367c0 -1 osd.2 28 log_to_monitors true 2024-09-16T15:27:51.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:27:52.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:27:52.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:27:52.990 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:27:52.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:27:52.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:27:52.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:27:53.421 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/689614850,v1:127.0.0.1:6811/689614850] [v2:127.0.0.1:6812/689614850,v1:127.0.0.1:6813/689614850] exists,up 42d6a639-404a-47f9-b44b-75f4fe86cfa5 2024-09-16T15:27:53.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:27:53.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:27:53.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:27:53.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:27:53.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-09-16T15:27:54.053 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already in. 2024-09-16T15:27:54.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:27:54.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-09-16T15:27:54.714 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already in. 2024-09-16T15:27:54.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-09-16T15:27:59.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-09-16T15:27:59.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-09-16T15:27:59.737 INFO:tasks.workunit.client.0.smithi039.stdout:writing non-divergent object existing_4 2024-09-16T15:27:59.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-09-16T15:27:59.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:28:00.153 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:28:00.153 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-16T15:27:55.032544+0000 28'100 41:120 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T21:32:14.112278+0000 0 0 2024-09-16T15:28:00.153 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:28:00.153 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:28:00.153 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:28:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-09-16T15:28:00.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T15:28:00.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-09-16T15:28:00.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:28:00.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:28:00.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:28:00.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T15:28:00.296 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:28:00.296 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:28:00.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:28:00.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:28:00.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:28:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T15:28:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:28:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:28:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:28:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:28:00.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:28:00.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:28:00.827 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:28:00.827 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:28:00.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:28:00.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:28:00.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:28:01.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=154618822661 2024-09-16T15:28:01.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 154618822661 2024-09-16T15:28:01.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822661' 2024-09-16T15:28:01.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:28:01.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:28:01.141 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T15:28:01.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T15:28:01.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T15:28:01.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T15:28:01.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:28:01.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:28:01.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=163208757252 2024-09-16T15:28:01.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 163208757252 2024-09-16T15:28:01.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822661 2-163208757252' 2024-09-16T15:28:01.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:28:01.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-154618822661 2024-09-16T15:28:01.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:28:01.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:28:01.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-154618822661 2024-09-16T15:28:01.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:28:01.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=154618822661 2024-09-16T15:28:01.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 154618822661' 2024-09-16T15:28:01.313 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 154618822661 2024-09-16T15:28:01.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:28:01.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 154618822659 -lt 154618822661 2024-09-16T15:28:01.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:28:02.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:28:02.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:28:03.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 154618822661 -lt 154618822661 2024-09-16T15:28:03.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:28:03.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-163208757252 2024-09-16T15:28:03.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:28:03.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:28:03.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-163208757252 2024-09-16T15:28:03.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:28:03.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=163208757252 2024-09-16T15:28:03.186 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 163208757252 2024-09-16T15:28:03.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 163208757252' 2024-09-16T15:28:03.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:28:03.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 163208757252 -lt 163208757252 2024-09-16T15:28:03.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:28:03.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:03.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:04.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:28:04.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:04.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:04.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:04.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:04.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:04.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:04.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:04.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:04.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:04.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:04.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:05.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:05.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T15:28:05.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T15:28:05.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T15:28:05.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:05.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T15:28:05.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:05.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:05.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:05.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:05.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:05.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:05.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:05.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:05.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:05.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:05.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:05.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:06.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:06.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:06.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:06.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:06.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:06.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:06.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:06.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:06.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:06.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:06.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:06.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-16T15:28:06.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:06.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T15:28:07.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:07.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:07.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:07.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:07.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:07.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:07.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:07.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:07.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:07.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:07.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:07.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:08.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:08.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:08.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:08.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:08.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:08.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:08.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:08.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:08.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:08.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:08.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:08.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-16T15:28:08.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:08.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T15:28:08.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:08.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:08.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:08.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:08.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:08.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:08.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:08.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:09.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:09.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:09.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:09.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:09.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:09.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:10.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:10.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:10.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-16T15:28:10.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:10.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-16T15:28:11.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:11.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:11.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:11.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:11.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:11.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:11.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:11.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:11.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:11.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:11.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:11.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:12.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:12.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:12.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:12.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:12.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:12.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:12.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:12.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:12.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:12.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:12.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:12.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-16T15:28:12.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:12.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-16T15:28:14.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:14.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:14.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:14.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:14.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:14.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:14.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:14.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:14.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:14.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:14.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:14.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:15.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:15.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:15.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:15.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:15.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:15.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:15.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:15.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:15.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:16.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:16.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:16.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-16T15:28:16.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:16.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-16T15:28:19.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:19.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:19.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:19.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:19.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:19.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:19.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:19.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:19.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:19.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:19.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:19.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:20.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:20.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:20.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:20.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:20.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:20.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:20.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:20.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:20.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:21.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:21.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:21.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-16T15:28:21.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:21.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-16T15:28:27.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:27.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:27.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:27.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:27.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:27.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:27.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:27.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:27.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:27.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:27.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:27.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:28.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:28.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:28.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:28.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:28.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:28.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:28.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:28.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:28.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:29.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:29.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:29.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-16T15:28:29.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:29.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-16T15:28:36.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:36.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:36.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:36.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:36.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:36.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:36.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:36.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:36.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:36.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:36.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:36.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:37.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:37.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:37.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:37.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:37.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:37.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:37.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:37.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:37.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:37.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:37.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:37.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-16T15:28:37.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T15:28:38.446 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:28:38.446 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "86902ecd-1180-4de0-903a-22b8052f9df3", 2024-09-16T15:28:38.446 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "18.2.4-845-ge4e227b3", 2024-09-16T15:28:38.446 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "e4e227b37858f5109138494e01e5ba1ba7b8ffe8", 2024-09-16T15:28:38.446 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-16T15:28:38.447328+0000", 2024-09-16T15:28:38.446 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-16T15:28:38.447 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-16T15:28:38.447 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-16T15:28:38.447 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-16T15:28:38.447 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-16T15:28:38.447 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:28:38.447 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:28:38.447 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-16T15:28:38.447 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-16T15:28:38.447 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.448 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-16T15:28:38.448 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:28:38.448 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.448 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-16T15:28:38.448 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:28:38.448 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:28:38.448 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-16T15:28:38.448 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:28:38.449 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.449 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:28:38.449 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.449 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-16T15:28:38.449 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.449 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.449 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:28:38.449 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.449 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-16T15:28:38.449 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:28:38.450 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:28:38.450 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-16T15:28:38.450 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:28:38.450 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.450 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:28:38.450 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.450 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-16T15:28:38.450 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.450 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.450 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:28:38.450 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.451 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-16T15:28:38.451 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:28:38.451 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:28:38.451 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-16T15:28:38.451 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:28:38.451 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.451 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:28:38.451 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.451 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-16T15:28:38.451 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.452 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.452 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-16T15:28:38.452 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.452 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.452 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:28:38.452 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.452 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.452 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-16T15:28:38.452 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.453 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-16T15:28:38.453 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-16T15:28:38.453 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-16T15:28:38.453 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:28:38.453 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:28:38.453 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:24:47.548252Z", 2024-09-16T15:28:38.453 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:24:47.548252Z", 2024-09-16T15:28:38.453 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 18, 2024-09-16T15:28:38.453 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "reef", 2024-09-16T15:28:38.453 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-16T15:28:38.454 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-16T15:28:38.454 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-16T15:28:38.454 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-16T15:28:38.454 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-16T15:28:38.454 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-16T15:28:38.454 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-16T15:28:38.454 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-16T15:28:38.454 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-16T15:28:38.454 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-16T15:28:38.454 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: "reef" 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.455 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-16T15:28:38.456 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-16T15:28:38.456 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:28:38.456 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.456 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.456 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.456 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-16T15:28:38.456 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-16T15:28:38.456 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.456 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.456 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.457 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-16T15:28:38.457 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-16T15:28:38.457 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-16T15:28:38.457 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-16T15:28:38.457 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-16T15:28:38.457 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.457 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.457 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.457 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-16T15:28:38.457 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-16T15:28:38.458 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.458 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-16T15:28:38.458 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 41, 2024-09-16T15:28:38.458 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:28:38.458 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:24:48.503410+0000", 2024-09-16T15:28:38.458 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:27:54.725784+0000", 2024-09-16T15:28:38.458 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-16T15:27:52.140176+0000", 2024-09-16T15:28:38.458 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-16T15:27:54.444955+0000", 2024-09-16T15:28:38.458 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T15:28:38.458 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-16T15:28:38.458 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-16T15:28:38.459 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-16T15:28:38.459 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-16T15:28:38.459 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-16T15:28:38.459 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-16T15:28:38.459 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-16T15:28:38.459 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-16T15:28:38.459 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-16T15:28:38.459 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.459 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 10, 2024-09-16T15:28:38.459 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T15:28:38.459 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T15:28:38.460 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T15:28:38.460 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-16T15:28:38.460 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-16T15:28:38.460 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-16T15:28:38.460 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-16T15:28:38.460 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-16T15:28:38.460 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "reef", 2024-09-16T15:28:38.460 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-16T15:28:38.460 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-16T15:28:38.461 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.461 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:28:38.461 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-16T15:28:38.461 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-16T15:26:41.209362+0000", 2024-09-16T15:28:38.461 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 16389, 2024-09-16T15:28:38.461 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-16T15:28:38.461 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 3, 2024-09-16T15:28:38.461 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-16T15:28:38.462 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-16T15:28:38.462 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 1, 2024-09-16T15:28:38.462 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-16T15:28:38.462 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-16T15:28:38.462 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T15:28:38.462 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T15:28:38.462 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-16T15:28:38.462 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-16T15:28:38.463 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 1, 2024-09-16T15:28:38.463 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 1, 2024-09-16T15:28:38.463 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-16T15:28:38.463 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 1, 2024-09-16T15:28:38.463 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 1, 2024-09-16T15:28:38.463 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-16T15:28:38.463 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-16T15:28:38.463 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-16T15:28:38.463 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-16T15:28:38.463 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-16T15:28:38.463 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-16T15:28:38.464 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-16T15:28:38.464 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.464 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "29", 2024-09-16T15:28:38.464 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-16T15:28:38.464 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T15:28:38.464 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T15:28:38.464 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-16T15:28:38.464 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "pool", 2024-09-16T15:28:38.464 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 1, 2024-09-16T15:28:38.464 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 29, 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [ 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: "snapid": 1, 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "2024-09-16T15:27:29.795538+0000", 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap1" 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-16T15:28:38.465 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "myprofile", 2024-09-16T15:28:38.466 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-16T15:28:38.467 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-16T15:28:38.467 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.467 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-16T15:28:38.467 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-16T15:28:38.467 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-16T15:28:38.467 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-16T15:28:38.467 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-16T15:28:38.467 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T15:28:38.467 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-16T15:28:38.467 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 8192, 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {} 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "eb303f60-3b45-4945-a069-9f2f1ab9cc77", 2024-09-16T15:28:38.468 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 28, 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 36, 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 40, 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 30, 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.469 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.470 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.470 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-16T15:28:38.470 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3891233128 2024-09-16T15:28:38.470 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.470 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.470 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.470 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-16T15:28:38.470 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3891233128 2024-09-16T15:28:38.470 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.470 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.470 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.471 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:28:38.471 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.471 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.471 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.471 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-16T15:28:38.471 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3891233128 2024-09-16T15:28:38.471 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.472 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.472 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.472 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-16T15:28:38.472 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3891233128 2024-09-16T15:28:38.472 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.472 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.472 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.472 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:28:38.472 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.472 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.472 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3891233128 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3891233128 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:28:38.473 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.474 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.474 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.474 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-16T15:28:38.474 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3891233128 2024-09-16T15:28:38.474 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.474 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.474 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.474 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-16T15:28:38.474 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3891233128 2024-09-16T15:28:38.474 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.474 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/3891233128", 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/3891233128", 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3891233128", 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3891233128", 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:28:38.475 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "409fbc6b-a686-4358-b419-40a385379580", 2024-09-16T15:28:38.476 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-16T15:28:38.476 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:28:38.476 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:28:38.476 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:28:38.476 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-16T15:28:38.476 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-16T15:28:38.476 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-16T15:28:38.476 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 24, 2024-09-16T15:28:38.476 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 31, 2024-09-16T15:28:38.476 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:28:38.476 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:28:38.477 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.477 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.477 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.477 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:28:38.477 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:28:38.477 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.477 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.477 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.477 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:28:38.477 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:28:38.477 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.478 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.478 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.478 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:28:38.478 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.478 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.478 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.478 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:28:38.478 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:28:38.478 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.478 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.479 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.479 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:28:38.479 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:28:38.479 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.479 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.479 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.479 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:28:38.479 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.479 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.479 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.479 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.480 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.481 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.481 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:28:38.481 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:28:38.481 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.481 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.481 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.481 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:28:38.481 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:28:38.481 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.481 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.482 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.482 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/1133638368", 2024-09-16T15:28:38.482 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/1133638368", 2024-09-16T15:28:38.482 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1133638368", 2024-09-16T15:28:38.482 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1133638368", 2024-09-16T15:28:38.482 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:28:38.482 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-16T15:28:38.482 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.482 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.482 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.482 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:28:38.483 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "42d6a639-404a-47f9-b44b-75f4fe86cfa5", 2024-09-16T15:28:38.483 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:28:38.483 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:28:38.483 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:28:38.483 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:28:38.483 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-16T15:28:38.483 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 28, 2024-09-16T15:28:38.483 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 38, 2024-09-16T15:28:38.483 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 0, 2024-09-16T15:28:38.484 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 32, 2024-09-16T15:28:38.484 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:28:38.484 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:28:38.484 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.484 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.484 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.484 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:28:38.484 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 689614850 2024-09-16T15:28:38.484 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 689614850 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.485 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 689614850 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 689614850 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.486 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.487 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.487 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:28:38.487 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 689614850 2024-09-16T15:28:38.487 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.487 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.487 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.487 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:28:38.487 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 689614850 2024-09-16T15:28:38.487 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.487 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.487 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 689614850 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 689614850 2024-09-16T15:28:38.488 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/689614850", 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/689614850", 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/689614850", 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/689614850", 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.489 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:27:41.034786+0000", 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:25:23.600785+0000", 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.490 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:27:41.623277+0000", 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:25:59.768079+0000", 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:27:42.241265+0000", 2024-09-16T15:28:38.491 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:26:36.545635+0000", 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-16T15:28:38.492 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "myprofile": { 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-device-class": "", 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-failure-domain": "osd", 2024-09-16T15:28:38.493 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-root": "default", 2024-09-16T15:28:38.494 INFO:tasks.workunit.client.0.smithi039.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-16T15:28:38.494 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-16T15:28:38.494 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "1", 2024-09-16T15:28:38.494 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-16T15:28:38.494 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van", 2024-09-16T15:28:38.494 INFO:tasks.workunit.client.0.smithi039.stdout: "w": "8" 2024-09-16T15:28:38.494 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.494 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-16T15:28:38.495 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/3891233128,v1:127.0.0.1:6805/3891233128]", 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:28:38.496 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:28:38.497 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:24:55.482416Z", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:28:38.498 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/3891233128,v1:127.0.0.1:6803/3891233128]", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3891233128,v1:127.0.0.1:6809/3891233128]", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3891233128,v1:127.0.0.1:6807/3891233128]", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:28:38.499 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:28:38.500 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-16T15:28:38.500 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:28:38.500 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:28:38.500 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:28:38.500 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.500 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.500 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:28:38.500 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:28:38.500 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/1133638368,v1:127.0.0.1:6813/1133638368]", 2024-09-16T15:28:38.500 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:28:38.500 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:28:38.501 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:25:31.289202Z", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:28:38.502 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/1133638368,v1:127.0.0.1:6811/1133638368]", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1133638368,v1:127.0.0.1:6817/1133638368]", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1133638368,v1:127.0.0.1:6815/1133638368]", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:28:38.503 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:28:38.504 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/689614850,v1:127.0.0.1:6813/689614850]", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-16T15:28:38.505 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:26:07.749502Z", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:28:38.506 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/689614850,v1:127.0.0.1:6811/689614850]", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/689614850,v1:127.0.0.1:6817/689614850]", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/689614850,v1:127.0.0.1:6815/689614850]", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:28:38.507 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:28:38.508 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:28:38.508 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:28:38.508 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:28:38.508 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:28:38.508 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:28:38.508 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:28:38.508 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-16T15:28:38.508 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:28:38.508 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:28:38.508 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:28:38.508 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [] 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 36 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.509 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 38 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 41, 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-16T15:28:38.510 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.511 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-16T15:28:38.512 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-16T15:28:38.513 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-16T15:28:38.514 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.515 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:28:38.516 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:28:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:28:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.519 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:28:38.520 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:28:38.521 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-16T15:28:38.522 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.523 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 1, 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "test", 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 3, 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "set_chooseleaf_tries", 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 5 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "set_choose_tries", 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 100 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-16T15:28:38.524 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_indep", 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.525 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-16T15:28:38.526 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 1, 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-16T15:28:38.527 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2" 2024-09-16T15:28:38.528 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:28:38.529 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 1, 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62667, 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 1, 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:28:38.530 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:28:38.531 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:28:38.532 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:28:38.533 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 18, 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 18, 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 3, 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 3, 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:28:38.534 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83624, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2124, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489176, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036916224, 2024-09-16T15:28:38.535 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2174976, 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1533259, 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:28:38.536 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-16T15:28:38.537 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83624, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2124, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489176, 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:28:38.538 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036916224, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2174976, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1533259, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10399, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416929 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:28:38.539 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.540 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62667, 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 1, 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:28:38.541 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:28:38.542 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:28:38.543 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1241088, 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1241088, 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:28:38.544 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 18, 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 18, 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 3, 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 3, 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 154618822668 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.545 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672992 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 163208757260 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 1 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.546 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2147483647, 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.547 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.548 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 5, 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:27:38.108877+0000", 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 104, 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-16T15:28:38.549 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 407, 2024-09-16T15:28:38.550 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-16T15:28:38.550 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:28:38.550 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 558, 2024-09-16T15:28:38.550 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-16T15:28:38.550 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-16T15:28:38.550 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:28:38.550 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:28:38.551 INFO:tasks.workunit.client.0.smithi039.stderr:report 4223492812 2024-09-16T15:28:38.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T15:28:38.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:28:38.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-09-16T15:28:38.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:28:38.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:28:38.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:28:38.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:28:38.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:28:38.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:28:38.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-09-16T15:28:38.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-16T15:28:38.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T15:28:38.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-16T15:28:38.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T15:28:38.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-09-16T15:28:38.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-09-16T15:28:39.788 INFO:tasks.workunit.client.0.smithi039.stderr:PG '1.0' not found 2024-09-16T15:28:40.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-16T15:28:40.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-16T15:28:40.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:28:40.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:28:40.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:28:40.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:28:40.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-16T15:28:40.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:28:40.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:28:40.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:28:40.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:28:40.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-16T15:28:40.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-16T15:28:40.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:28:40.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:28:40.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:28:40.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:28:40.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:28:40.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:28:40.330 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:28:40.330 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:28:40.330 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:28:40.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:28:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:28:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:28:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:28:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:28:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:28:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:28:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:28:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:28:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:28:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:28:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-16T15:28:40.332 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:28:40.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:28:40.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:28:40.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-16T15:28:40.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:28:40.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:28:40.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:28:40.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:28:40.375 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:28:40.373+0000 7f2aa14fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:28:40.378 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:28:40.377+0000 7f2aa14fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:28:40.379 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:28:40.381+0000 7f2aa14fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:28:40.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:28:40.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:28:40.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:28:40.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:28:40.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:28:40.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:28:40.795 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:28:40.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:28:40.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:28:40.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:28:40.949 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:28:40.949+0000 7f2aa14fc7c0 -1 Falling back to public interface 2024-09-16T15:28:41.219 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/3891233128,v1:127.0.0.1:6803/3891233128] [v2:127.0.0.1:6804/3891233128,v1:127.0.0.1:6805/3891233128] exists,up eb303f60-3b45-4945-a069-9f2f1ab9cc77 2024-09-16T15:28:41.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:28:41.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:28:41.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:28:41.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:28:41.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-09-16T15:28:41.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:28:41.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:28:41.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:28:41.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:28:41.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:28:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:28:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-09-16T15:28:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-16T15:28:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T15:28:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-16T15:28:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T15:28:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-09-16T15:28:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-09-16T15:28:42.749 INFO:tasks.workunit.client.0.smithi039.stderr:PG '1.0' not found 2024-09-16T15:28:43.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-16T15:28:43.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-16T15:28:43.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:28:43.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:28:43.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:28:43.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:28:43.050 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:28:43.050 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:28:43.050 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:28:43.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:28:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:28:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:28:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:28:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:28:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:28:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:28:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:28:43.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:28:43.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:28:43.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:28:43.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-16T15:28:43.052 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:28:43.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T15:28:43.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:28:43.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-16T15:28:43.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T15:28:43.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:28:43.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:28:43.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:28:43.094 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:28:43.093+0000 7f6fabf7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:28:43.097 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:28:43.097+0000 7f6fabf7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:28:43.098 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:28:43.097+0000 7f6fabf7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:28:43.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T15:28:43.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:28:43.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:28:43.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:28:43.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:28:43.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:28:43.486 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:28:43.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:28:43.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:28:43.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:28:43.524 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:28:43.525+0000 7f2aa14fc7c0 -1 osd.0 41 log_to_monitors true 2024-09-16T15:28:43.657 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:28:43.657+0000 7f6fabf7b7c0 -1 Falling back to public interface 2024-09-16T15:28:43.929 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/689614850,v1:127.0.0.1:6811/689614850] [v2:127.0.0.1:6812/689614850,v1:127.0.0.1:6813/689614850] exists,up 42d6a639-404a-47f9-b44b-75f4fe86cfa5 2024-09-16T15:28:43.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:28:43.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:28:43.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:28:43.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-09-16T15:28:43.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-16T15:28:43.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T15:28:43.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T15:28:43.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T15:28:43.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-16T15:28:43.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-09-16T15:28:45.598 INFO:tasks.workunit.client.0.smithi039.stderr:PG '1.0' not found 2024-09-16T15:28:46.110 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:28:46.109+0000 7f6fabf7b7c0 -1 osd.2 41 log_to_monitors true 2024-09-16T15:28:46.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-16T15:28:46.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T15:28:46.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-09-16T15:28:46.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:28:46.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:28:46.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:28:46.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T15:28:46.169 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:28:46.169 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:28:46.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:28:46.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:28:46.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:28:46.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T15:28:46.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:28:46.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:28:46.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:28:46.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:28:46.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:28:46.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:28:46.763 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:28:46.763 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:28:46.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:28:46.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:28:46.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:28:46.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=188978561027 2024-09-16T15:28:46.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 188978561027 2024-09-16T15:28:46.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-188978561027' 2024-09-16T15:28:46.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:28:46.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:28:47.108 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T15:28:47.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T15:28:47.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T15:28:47.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T15:28:47.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:28:47.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:28:47.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=206158430211 2024-09-16T15:28:47.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 206158430211 2024-09-16T15:28:47.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-188978561027 2-206158430211' 2024-09-16T15:28:47.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:28:47.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-188978561027 2024-09-16T15:28:47.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:28:47.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:28:47.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-188978561027 2024-09-16T15:28:47.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:28:47.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=188978561027 2024-09-16T15:28:47.289 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 188978561027 2024-09-16T15:28:47.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 188978561027' 2024-09-16T15:28:47.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:28:47.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 188978561027 2024-09-16T15:28:47.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:28:48.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:28:48.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:28:49.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 188978561027 -lt 188978561027 2024-09-16T15:28:49.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:28:49.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-206158430211 2024-09-16T15:28:49.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:28:49.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:28:49.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:28:49.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-206158430211 2024-09-16T15:28:49.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=206158430211 2024-09-16T15:28:49.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 206158430211' 2024-09-16T15:28:49.200 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 206158430211 2024-09-16T15:28:49.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:28:49.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 206158430211 -lt 206158430211 2024-09-16T15:28:49.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:28:49.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:49.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:50.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:28:50.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:50.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:50.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:50.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:50.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:50.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:50.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:50.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:50.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:50.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:50.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:51.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:51.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T15:28:51.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T15:28:51.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T15:28:51.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:51.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T15:28:51.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:51.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:51.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:51.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:51.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:51.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:51.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:51.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:51.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:51.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:51.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:51.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:52.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:52.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:52.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:52.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:52.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:52.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:52.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:52.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:52.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:52.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:52.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:52.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-16T15:28:52.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:52.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T15:28:53.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:53.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:53.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:53.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:53.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:53.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:53.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:53.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:53.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:53.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:53.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:53.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:54.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:54.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:54.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:54.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:54.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:54.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:54.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:54.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:54.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:54.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:54.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:54.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-16T15:28:54.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:54.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T15:28:54.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:54.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:54.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:54.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:54.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:54.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:54.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:54.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:55.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:55.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:55.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:55.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:56.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:56.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:56.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:56.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:56.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:56.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:56.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:56.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:56.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:56.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:56.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:56.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-16T15:28:56.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:56.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-16T15:28:57.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:28:57.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:28:57.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:28:57.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:28:57.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:28:57.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:28:57.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:28:57.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:28:57.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:28:57.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:28:57.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:28:57.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:28:58.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:28:58.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:28:58.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:28:58.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:28:58.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:28:58.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:28:58.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:28:58.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:28:58.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:28:58.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:28:58.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:28:58.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-16T15:28:58.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:28:58.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-16T15:29:00.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:29:00.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:29:00.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:29:00.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:29:00.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:29:00.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:29:00.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:29:00.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:29:00.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:29:00.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:29:00.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:29:00.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:29:01.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:29:01.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:29:01.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:29:01.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:29:01.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:29:01.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:29:01.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:29:01.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:29:01.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:29:02.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:29:02.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:29:02.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-16T15:29:02.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:29:02.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-16T15:29:05.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:29:05.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:29:05.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:29:05.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:29:05.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:29:05.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:29:05.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:29:05.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:29:05.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:29:05.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:29:05.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:29:05.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:29:06.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:29:06.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:29:06.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:29:06.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:29:06.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:29:06.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:29:06.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:29:06.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:29:06.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:29:06.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:29:06.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:29:06.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-16T15:29:06.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:29:06.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-16T15:29:13.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:29:13.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:29:13.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:29:13.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:29:13.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:29:13.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:29:13.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:29:13.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:29:13.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:29:13.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:29:13.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:29:13.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:29:14.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:29:14.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:29:14.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:29:14.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:29:14.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:29:14.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:29:14.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:29:14.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:29:14.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:29:14.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:29:14.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:29:14.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-16T15:29:14.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T15:29:14.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-16T15:29:22.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T15:29:22.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:29:22.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:29:22.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:29:22.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:29:22.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:29:22.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:29:22.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:29:22.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T15:29:22.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:29:22.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:29:22.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:29:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T15:29:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T15:29:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T15:29:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T15:29:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T15:29:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T15:29:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T15:29:23.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T15:29:23.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T15:29:23.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T15:29:23.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T15:29:23.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-16T15:29:23.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T15:29:24.160 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:29:24.160 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "86902ecd-1180-4de0-903a-22b8052f9df3", 2024-09-16T15:29:24.160 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "18.2.4-845-ge4e227b3", 2024-09-16T15:29:24.160 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "e4e227b37858f5109138494e01e5ba1ba7b8ffe8", 2024-09-16T15:29:24.160 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-16T15:29:24.161642+0000", 2024-09-16T15:29:24.160 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-16T15:29:24.160 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-16T15:29:24.160 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:29:24.161 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-16T15:29:24.162 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T15:29:24.163 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-16T15:29:24.164 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:24:47.548252Z", 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:24:47.548252Z", 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 18, 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "reef", 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-16T15:29:24.165 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: "reef" 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-16T15:29:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-16T15:29:24.167 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 49, 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "710062ef-0f1c-422d-8388-4a3c4e7277dd", 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T15:24:48.503410+0000", 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:28:47.107440+0000", 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-16T15:28:46.738944+0000", 2024-09-16T15:29:24.168 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-16T15:27:54.444955+0000", 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 12, 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T15:29:24.169 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "reef", 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-16T15:26:41.209362+0000", 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 16389, 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-16T15:29:24.170 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 3, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 1, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 1, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 1, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-16T15:29:24.171 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 1, 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 1, 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "29", 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T15:29:24.172 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "pool", 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 1, 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 29, 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [ 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "snapid": 1, 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "2024-09-16T15:27:29.795538+0000", 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap1" 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-16T15:29:24.173 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "myprofile", 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.174 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 8192, 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {} 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.175 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "eb303f60-3b45-4945-a069-9f2f1ab9cc77", 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 36, 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 41, 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 44, 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 48, 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 43, 2024-09-16T15:29:24.176 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 487489081 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 487489081 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 487489081 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 487489081 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.178 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 487489081 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 487489081 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.179 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 487489081 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 487489081 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/487489081", 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/487489081", 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/487489081", 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/487489081", 2024-09-16T15:29:24.180 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "409fbc6b-a686-4358-b419-40a385379580", 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-16T15:29:24.181 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 24, 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 31, 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.183 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.184 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1133638368 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/1133638368", 2024-09-16T15:29:24.185 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/1133638368", 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1133638368", 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1133638368", 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "42d6a639-404a-47f9-b44b-75f4fe86cfa5", 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T15:29:24.186 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 38, 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 41, 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 48, 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 0, 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 47, 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3377558763 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.187 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3377558763 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3377558763 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3377558763 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3377558763 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T15:29:24.189 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3377558763 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3377558763 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T15:29:24.190 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3377558763 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/3377558763", 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/3377558763", 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3377558763", 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3377558763", 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.191 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:28:43.860336+0000", 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:25:23.600785+0000", 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.192 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:27:41.623277+0000", 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:25:59.768079+0000", 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-16T15:28:46.494248+0000", 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T15:29:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T15:26:36.545635+0000", 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-16T15:29:24.194 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "myprofile": { 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-device-class": "", 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-failure-domain": "osd", 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-root": "default", 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "1", 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van", 2024-09-16T15:29:24.195 INFO:tasks.workunit.client.0.smithi039.stdout: "w": "8" 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-16T15:29:24.196 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/487489081,v1:127.0.0.1:6805/487489081]", 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:29:24.197 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:29:24.198 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:24:55.482416Z", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/487489081,v1:127.0.0.1:6803/487489081]", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/487489081,v1:127.0.0.1:6809/487489081]", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/487489081,v1:127.0.0.1:6807/487489081]", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:29:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:29:24.200 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/1133638368,v1:127.0.0.1:6813/1133638368]", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:29:24.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:25:31.289202Z", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:29:24.202 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/1133638368,v1:127.0.0.1:6811/1133638368]", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1133638368,v1:127.0.0.1:6817/1133638368]", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1133638368,v1:127.0.0.1:6815/1133638368]", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:29:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/3377558763,v1:127.0.0.1:6813/3377558763]", 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T15:29:24.204 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T15:29:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T15:26:07.749502Z", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/3377558763,v1:127.0.0.1:6811/3377558763]", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3377558763,v1:127.0.0.1:6817/3377558763]", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3377558763,v1:127.0.0.1:6815/3377558763]", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T15:29:24.206 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 25, 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-16T15:29:24.207 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [ 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: "floor": 25 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 44 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.208 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:29:24.209 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 48 2024-09-16T15:29:24.209 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.209 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.209 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.209 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-16T15:29:24.209 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 49, 2024-09-16T15:29:24.209 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-16T15:29:24.209 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-16T15:29:24.209 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.209 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:29:24.210 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-16T15:29:24.210 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:29:24.210 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.210 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.210 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:29:24.210 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-16T15:29:24.210 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:29:24.210 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:29:24.211 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.212 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-16T15:29:24.213 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-16T15:29:24.213 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.213 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.213 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-16T15:29:24.213 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-16T15:29:24.213 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.213 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.213 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-16T15:29:24.213 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-16T15:29:24.213 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.213 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-16T15:29:24.214 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:29:24.215 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:29:24.216 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:29:24.217 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.218 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:29:24.218 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:29:24.218 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:29:24.218 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.218 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.218 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:29:24.218 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:29:24.218 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:29:24.218 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.218 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.218 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T15:29:24.219 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T15:29:24.220 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.221 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 1, 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "test", 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 3, 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.222 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "set_chooseleaf_tries", 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 5 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "set_choose_tries", 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 100 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_indep", 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-16T15:29:24.223 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-16T15:29:24.224 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.224 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.224 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-16T15:29:24.224 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.224 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T15:29:24.224 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.224 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.224 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-16T15:29:24.224 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-16T15:29:24.224 INFO:tasks.workunit.client.0.smithi039.stderr:report 2127368872 2024-09-16T15:29:24.225 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-16T15:29:24.225 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-16T15:29:24.225 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-16T15:29:24.225 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-16T15:29:24.225 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-16T15:29:24.225 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-16T15:29:24.225 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-16T15:29:24.225 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-16T15:29:24.225 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-16T15:29:24.225 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-16T15:29:24.225 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 1, 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T15:29:24.226 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2" 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.227 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-16T15:29:24.228 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 1, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62667, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 1, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:29:24.229 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:29:24.230 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:29:24.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:29:24.232 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 18, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 18, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 3, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 3, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:29:24.233 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 85056, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2252, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 15, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 82736, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314487744, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322035449856, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2306048, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1586983, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 16272, 2024-09-16T15:29:24.234 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 84721776 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T15:29:24.235 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-16T15:29:24.236 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 85056, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2252, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 15, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 82736, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314487744, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322035449856, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2306048, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1586983, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:29:24.237 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 16272, 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 84721776 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T15:29:24.238 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62667, 2024-09-16T15:29:24.239 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 1, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-16T15:29:24.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T15:29:24.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T15:29:24.242 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1241088, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1241088, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 18, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 18, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 3, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 3, 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-16T15:29:24.243 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 188978561034 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672992 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 206158430218 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.244 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 1 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2147483647, 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.245 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-16T15:29:24.246 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 5, 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T15:27:38.108877+0000", 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-16T15:29:24.247 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 126, 2024-09-16T15:29:24.248 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-16T15:29:24.248 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 489, 2024-09-16T15:29:24.248 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-16T15:29:24.248 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T15:29:24.248 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 668, 2024-09-16T15:29:24.248 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-16T15:29:24.248 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-16T15:29:24.248 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T15:29:24.248 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:29:24.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T15:29:24.249 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery 2024-09-16T15:29:24.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-09-16T15:29:24.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:29:24.638 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:29:24.639 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-16T15:28:47.426391+0000 41'102 49:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:37:15.651476+0000 0 0 2024-09-16T15:29:24.639 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:29:24.639 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:29:24.639 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:29:24.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:29:24.657 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-09-16T15:29:24.657 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T15:29:24.657 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T15:29:24.657 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:29:24.657 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:29:24.658 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:29:24.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.0.asok 2024-09-16T15:29:24.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-16T15:29:24.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.0.asok set_recovery_delay 100000 2024-09-16T15:29:24.774 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:29:24.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-16T15:29:24.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-09-16T15:29:24.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T15:29:24.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T15:29:24.789 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:29:24.789 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:29:24.789 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:29:24.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.2.asok 2024-09-16T15:29:24.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-16T15:29:24.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.2.asok set_recovery_delay 100000 2024-09-16T15:29:24.905 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:29:24.918 INFO:tasks.workunit.client.0.smithi039.stdout:revive divergent 1 2024-09-16T15:29:24.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-09-16T15:29:24.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:29:25.322 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:29:25.322 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-16T15:28:47.426391+0000 41'102 49:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:37:15.651476+0000 0 0 2024-09-16T15:29:25.322 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:29:25.322 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:29:25.322 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:29:25.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-09-16T15:29:26.109 INFO:tasks.workunit.client.0.smithi039.stderr:noup is set 2024-09-16T15:29:26.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-16T15:29:26.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:29:26.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:29:26.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:29:26.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:29:26.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:29:26.136 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:29:26.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:29:26.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:29:26.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-16T15:29:26.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:29:26.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:29:26.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:29:26.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:29:26.178 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:29:26.176+0000 7f12be41e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:29:26.180 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:29:26.180+0000 7f12be41e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:29:26.182 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:29:26.180+0000 7f12be41e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:29:26.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-16T15:29:26.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-09-16T15:29:26.725 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:29:26.724+0000 7f12be41e7c0 -1 Falling back to public interface 2024-09-16T15:29:29.298 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:29:29.296+0000 7f12be41e7c0 -1 osd.1 29 log_to_monitors true 2024-09-16T15:29:31.585 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery divergent 2024-09-16T15:29:31.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-09-16T15:29:31.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:29:32.000 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:29:32.001 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-16T15:28:47.426391+0000 41'102 49:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:37:15.651476+0000 0 0 2024-09-16T15:29:32.001 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:29:32.001 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:29:32.001 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:29:32.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-09-16T15:29:32.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T15:29:32.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T15:29:32.018 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:29:32.018 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:29:32.018 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:29:32.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27113/ceph-osd.1.asok 2024-09-16T15:29:32.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-09-16T15:29:32.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27113/ceph-osd.1.asok set_recovery_delay 100000 2024-09-16T15:29:32.142 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-16T15:29:32.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-09-16T15:29:32.938 INFO:tasks.workunit.client.0.smithi039.stderr:noup is unset 2024-09-16T15:29:32.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-09-16T15:29:32.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:29:32.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:29:32.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:29:32.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:29:32.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:29:32.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:29:32.959 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:29:32.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:29:32.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:29:33.639 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 44 up_thru 48 down_at 43 last_clean_interval [36,41) [v2:127.0.0.1:6802/487489081,v1:127.0.0.1:6803/487489081] [v2:127.0.0.1:6804/487489081,v1:127.0.0.1:6805/487489081] exists,up eb303f60-3b45-4945-a069-9f2f1ab9cc77 2024-09-16T15:29:33.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:29:33.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:29:33.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:29:33.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-09-16T15:29:33.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:29:33.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:29:33.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:29:33.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:29:33.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:29:33.641 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:29:33.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:29:33.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:29:33.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:29:34.187 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 54 up_thru 54 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1160975082,v1:127.0.0.1:6819/1160975082] [v2:127.0.0.1:6820/1160975082,v1:127.0.0.1:6821/1160975082] exists,up 409fbc6b-a686-4358-b419-40a385379580 2024-09-16T15:29:34.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:29:34.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:29:34.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:29:34.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-09-16T15:29:34.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:29:34.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:29:34.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:29:34.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:29:34.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:29:34.190 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:29:34.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:29:34.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:29:34.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:29:34.627 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 48 up_thru 0 down_at 47 last_clean_interval [38,41) [v2:127.0.0.1:6810/3377558763,v1:127.0.0.1:6811/3377558763] [v2:127.0.0.1:6812/3377558763,v1:127.0.0.1:6813/3377558763] exists,up 42d6a639-404a-47f9-b44b-75f4fe86cfa5 2024-09-16T15:29:34.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:29:34.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:29:34.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:29:34.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:29:35.035 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:29:35.035 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-16T15:28:47.426391+0000 41'102 49:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:37:15.651476+0000 0 0 2024-09-16T15:29:35.036 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:29:35.036 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:29:35.036 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:29:35.051 INFO:tasks.workunit.client.0.smithi039.stdout:wait for peering 2024-09-16T15:29:35.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-09-16T15:29:35.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:29:35.466 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:29:35.467 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-16T15:28:47.426391+0000 41'102 49:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:37:15.651476+0000 0 0 2024-09-16T15:29:35.467 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:29:35.467 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:29:35.467 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:29:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-09-16T15:29:35.630 INFO:tasks.workunit.client.0.smithi039.stdout:killing divergent 1 2024-09-16T15:29:35.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-09-16T15:29:35.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:29:36.043 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:29:36.043 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-16T15:28:47.426391+0000 41'102 49:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:37:15.651476+0000 0 0 2024-09-16T15:29:36.044 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:29:36.044 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:29:36.044 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:29:36.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-09-16T15:29:36.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:29:36.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:29:36.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:29:36.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:29:36.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:29:36.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:29:36.175 INFO:tasks.workunit.client.0.smithi039.stdout:reviving divergent 1 2024-09-16T15:29:36.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-09-16T15:29:36.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:29:36.642 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:29:36.643 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-09-16T15:29:33.945933+0000 41'102 55:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:57:10.334249+0000 0 0 2024-09-16T15:29:36.643 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:29:36.643 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:29:36.643 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:29:36.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-16T15:29:36.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-16T15:29:36.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:29:36.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:29:36.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:29:36.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-16T15:29:36.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T15:29:36.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:29:36.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:29:36.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:29:36.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-16T15:29:36.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-16T15:29:36.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:29:36.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:29:36.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-16T15:29:36.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:29:36.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:29:36.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:29:36.665 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:29:36.665 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:29:36.665 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:29:36.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:29:36.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' 2024-09-16T15:29:36.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:29:36.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-16T15:29:36.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-16T15:29:36.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:29:36.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:29:36.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:29:36.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:29:36.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:29:36.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:29:36.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-16T15:29:36.666 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:29:36.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:29:36.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=710062ef-0f1c-422d-8388-4a3c4e7277dd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27113/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:29:36.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-16T15:29:36.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:29:36.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:29:36.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:29:36.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:29:36.700 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:29:36.700+0000 7f6bfd34a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:29:36.702 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:29:36.700+0000 7f6bfd34a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:29:36.703 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:29:36.700+0000 7f6bfd34a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:29:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T15:29:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:29:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:29:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:29:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:29:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:29:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:29:37.093 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:29:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:29:37.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:29:37.257 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:29:37.256+0000 7f6bfd34a7c0 -1 Falling back to public interface 2024-09-16T15:29:37.510 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 54 up_thru 54 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1160975082,v1:127.0.0.1:6819/1160975082] [v2:127.0.0.1:6820/1160975082,v1:127.0.0.1:6821/1160975082] exists,up 409fbc6b-a686-4358-b419-40a385379580 2024-09-16T15:29:37.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:29:37.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:29:37.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:29:37.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-09-16T15:29:40.166 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:29:40.164+0000 7f6bfd34a7c0 -1 osd.1 55 log_to_monitors true 2024-09-16T15:29:57.512 INFO:tasks.workunit.client.0.smithi039.stdout:allowing recovery 2024-09-16T15:29:57.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-09-16T15:29:57.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:29:57.934 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:29:57.934 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-09-16T15:29:42.217859+0000 55'103 60:152 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-18T00:13:18.449904+0000 0 0 2024-09-16T15:29:57.934 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:29:57.934 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:29:57.934 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:29:57.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-16T15:29:57.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-16T15:29:58.102 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:29:58.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-16T15:29:58.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-16T15:29:58.274 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:29:58.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-16T15:29:58.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-16T15:29:58.440 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T15:29:58.455 INFO:tasks.workunit.client.0.smithi039.stdout:reading divergent objects 2024-09-16T15:29:58.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-09-16T15:29:58.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-09-16T15:29:58.851 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T15:29:58.851 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-09-16T15:29:42.217859+0000 55'103 60:152 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T15:26:41.351992+0000 0'0 2024-09-16T15:26:41.351992+0000 0 0 periodic scrub scheduled @ 2024-09-18T00:13:18.449904+0000 0 0 2024-09-16T15:29:58.851 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T15:29:58.851 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T15:29:58.851 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T15:29:58.867 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-09-16T15:29:58.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-09-16T15:29:58.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:29:58.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-09-16T15:29:58.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:29:58.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-09-16T15:29:58.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:29:58.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-09-16T15:29:58.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-16T15:29:58.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-09-16T15:29:59.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-09-16T15:29:59.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-16T15:29:59.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-16T15:29:59.030 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-16T15:27:21.926225+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-16T15:29:59.030 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 20 _merge_object_divergent_entries: keeping 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-16T15:27:21.926225+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-16T15:29:59.030 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:29:59.030 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 28'99 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-16T15:29:59.030 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-09-16T15:29:59.030 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-09-16T15:29:59.030 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-16T15:27:30.133437+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2024-09-16T15:29:59.030 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 20 _merge_object_divergent_entries: keeping 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-16T15:27:30.133437+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2024-09-16T15:29:59.030 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:29:59.030 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 28'99 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-16T15:29:59.030 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-09-16T15:29:59.031 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-09-16T15:29:59.031 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 29'104 2024-09-16T15:29:59.031 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-16T15:27:30.133437+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2024-09-16T15:29:59.031 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-09-16T15:29:59.031 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4646.0:1 2024-09-16T15:27:28.789283+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0,28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-16T15:27:29.795862+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:29:59.031 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4646.0:1 2024-09-16T15:27:28.789283+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:29:59.032 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 20 _merge_object_divergent_entries: keeping 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-16T15:27:29.795862+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:29:59.032 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-16T15:29:59.032 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'102 2024-09-16T15:29:59.032 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-16T15:29:59.032 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-16T15:29:59.032 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'104 2024-09-16T15:29:59.032 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-16T15:27:29.795862+0000 0 ObjectCleanRegions clean_offsets: [967~18446744073709550648], clean_omap: 1, new_object: 0 2024-09-16T15:29:59.032 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-16T15:29:33.788+0000 7f12a23a9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-16T15:29:59.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-16T15:29:59.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-16T15:29:59.039 INFO:tasks.workunit.client.0.smithi039.stdout:success 2024-09-16T15:29:59.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-09-16T15:29:59.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-09-16T15:29:59.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T15:29:59.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T15:29:59.614 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T15:29:59.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-09-16T15:29:59.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:29:59.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:29:59.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:29:59.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:29:59.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:30:04.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:30:04.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-16T15:30:04.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-16T15:30:04.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:30:04.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-16T15:30:04.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:30:04.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:30:04.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:30:04.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:30:04.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:30:04.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:30:04.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:30:04.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:30:04.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:30:04.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:30:04.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:30:04.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:30:04.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:30:04.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:30:04.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:30:04.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:30:04.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:30:04.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:30:04.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-16T15:30:04.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:30:04.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:30:04.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:30:04.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27113 2024-09-16T15:30:04.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:30:04.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:30:04.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T15:30:04.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/divergent-priors 0 2024-09-16T15:30:04.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-16T15:30:04.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T15:30:04.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-16T15:30:04.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:30:04.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:30:04.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:30:04.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:30:04.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:30:05.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:30:05.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:30:05.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:30:05.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:30:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:30:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:30:05.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:30:05.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:30:05.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:30:05.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:30:05.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:30:05.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:30:05.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T15:30:05.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-16T15:30:05.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:30:05.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:30:05.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27113 2024-09-16T15:30:05.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27113 2024-09-16T15:30:05.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:30:05.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:30:05.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T15:30:05.011 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T15:30:05.011 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T15:30:05.024 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-09-16T15:30:05.024 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-09-16T15:30:05.078 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T15:30:05.083 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T15:30:05.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:30:05.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:30:05.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T15:30:05.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T15:30:05.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T15:30:05.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T15:30:05.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T15:30:05.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T15:30:05.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/ec-error-rollforward 2024-09-16T15:30:05.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-09-16T15:30:05.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-09-16T15:30:05.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-09-16T15:30:05.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-09-16T15:30:05.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-09-16T15:30:05.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-09-16T15:30:05.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none ' 2024-09-16T15:30:05.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-09-16T15:30:05.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-09-16T15:30:05.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-09-16T15:30:05.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-09-16T15:30:05.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-09-16T15:30:05.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-09-16T15:30:05.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-09-16T15:30:05.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-09-16T15:30:05.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T15:30:05.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-09-16T15:30:05.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-09-16T15:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-09-16T15:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2024-09-16T15:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2024-09-16T15:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-16T15:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-16T15:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:30:05.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:30:05.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:30:05.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:30:05.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:30:05.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:30:05.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:30:05.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:30:05.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:30:05.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:30:05.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:30:05.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:30:05.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:30:05.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:30:05.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-16T15:30:05.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:30:05.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:30:05.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:30:05.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.84027 2024-09-16T15:30:05.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:30:05.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:30:05.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2024-09-16T15:30:05.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:30:05.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:30:05.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:30:05.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.84027 2024-09-16T15:30:05.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:30:05.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T15:30:05.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T15:30:05.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:30:05.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-09-16T15:30:05.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-09-16T15:30:05.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-09-16T15:30:05.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-09-16T15:30:05.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2024-09-16T15:30:05.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:30:05.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:30:05.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:30:05.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2024-09-16T15:30:05.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-09-16T15:30:05.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:30:05.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:30:05.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:30:05.498 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:30:05.498 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:30:05.498 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:30:05.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:30:05.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:30:05.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:30:05.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:30:05.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:30:05.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:30:05.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:30:05.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:30:05.710 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:30:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:30:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:30:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:30:05.712 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:30:05.712 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:30:05.712 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.84027/ceph-mon.a.asok 2024-09-16T15:30:05.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:30:05.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.84027/ceph-mon.a.asok config get fsid 2024-09-16T15:30:05.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:30:05.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:30:05.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:30:05.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:30:05.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:30:05.835 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:30:05.835 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:30:05.835 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:30:05.835 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:30:05.835 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:30:05.836 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:30:05.836 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.84027/ceph-mon.a.asok 2024-09-16T15:30:05.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:30:05.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.84027/ceph-mon.a.asok config get mon_host 2024-09-16T15:30:05.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-09-16T15:30:05.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2024-09-16T15:30:05.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:30:05.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:30:05.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:30:05.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2024-09-16T15:30:05.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:30:06.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:30:06.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:30:06.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:30:06.886 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:30:06.886 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:30:06.886 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:30:06.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:30:06.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:30:06.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:30:06.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-09-16T15:30:06.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-16T15:30:06.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:30:06.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:30:06.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:30:06.958 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:30:06.959 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:30:06.959 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:30:06.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:30:06.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:30:06.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:30:06.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:30:06.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:30:06.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-16T15:30:06.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-16T15:30:06.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:30:06.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:30:06.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:30:06.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:30:06.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:30:06.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:30:06.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2024-09-16T15:30:06.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:30:06.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=91501509-0cc7-468c-8073-f1575b15d39c 2024-09-16T15:30:06.961 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 91501509-0cc7-468c-8073-f1575b15d39c 2024-09-16T15:30:06.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 91501509-0cc7-468c-8073-f1575b15d39c' 2024-09-16T15:30:06.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:30:06.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+TuhmKNlhOhAA0tgCaX6MJMr8lvqHpCkbQg== 2024-09-16T15:30:06.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+TuhmKNlhOhAA0tgCaX6MJMr8lvqHpCkbQg=="}' 2024-09-16T15:30:06.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 91501509-0cc7-468c-8073-f1575b15d39c -i td/ec-error-rollforward/0/new.json 2024-09-16T15:30:07.295 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:30:07.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2024-09-16T15:30:07.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+TuhmKNlhOhAA0tgCaX6MJMr8lvqHpCkbQg== --osd-uuid 91501509-0cc7-468c-8073-f1575b15d39c 2024-09-16T15:30:07.338 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:07.335+0000 7f02bf3fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:07.340 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:07.339+0000 7f02bf3fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:07.341 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:07.339+0000 7f02bf3fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:07.342 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:07.339+0000 7f02bf3fb7c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-09-16T15:30:07.342 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:07.339+0000 7f02bf3fb7c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-09-16T15:30:07.342 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:07.339+0000 7f02bf3fb7c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-09-16T15:30:07.379 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:07.375+0000 7f02bf3fb7c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-09-16T15:30:13.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-09-16T15:30:13.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:30:13.781 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:30:13.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:30:13.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:30:14.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:30:14.349 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:30:14.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:30:14.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:30:14.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:30:14.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:30:14.377 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:14.375+0000 7f4892ea57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:14.405 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:14.403+0000 7f4892ea57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:14.407 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:14.407+0000 7f4892ea57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:14.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:30:14.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:30:14.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:30:14.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:30:14.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:30:14.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:14.774 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:30:14.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:30:14.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:14.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:14.978 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:14.975+0000 7f4892ea57c0 -1 Falling back to public interface 2024-09-16T15:30:15.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:16.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:16.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:16.239 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:30:16.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:30:16.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:16.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:16.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:17.573 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:17.571+0000 7f4892ea57c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:30:17.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:17.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:17.655 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:30:17.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:30:17.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:17.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:18.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:19.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:19.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:19.195 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:30:19.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:30:19.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:19.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:19.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:20.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:20.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:20.659 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:30:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:30:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:20.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:21.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:22.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:22.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:22.111 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:30:22.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:30:22.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:22.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:22.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:23.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:23.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:23.713 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:30:23.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:30:23.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:23.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:24.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:25.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:25.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:25.145 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:30:25.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:30:25.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:25.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:25.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:26.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:26.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:26.685 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:30:26.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:30:26.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:26.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:27.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:28.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:28.129 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:30:28.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:30:28.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:28.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:29.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:29.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:29.665 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:30:29.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:30:29.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:29.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:30.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:31.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:31.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:31.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:30:31.373 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:30:31.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:31.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:31.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:32.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:32.957 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:30:32.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:32.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:30:32.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:32.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:33.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:34.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:34.788 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:30:34.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:34.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:30:34.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:34.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:35.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:36.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:36.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:36.521 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:30:36.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:30:36.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:36.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:37.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:38.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:38.319 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:30:38.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:38.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:30:38.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:38.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:39.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:40.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:40.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:40.345 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:30:40.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:30:40.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:40.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:41.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:42.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:42.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:42.081 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:30:42.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:30:42.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:42.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:30:42.513 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/293590132,v1:127.0.0.1:6803/293590132] [v2:127.0.0.1:6804/293590132,v1:127.0.0.1:6805/293590132] exists,up 91501509-0cc7-468c-8073-f1575b15d39c 2024-09-16T15:30:42.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:30:42.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:30:42.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:30:42.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-09-16T15:30:42.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-16T15:30:42.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:30:42.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:30:42.517 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:30:42.517 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:30:42.517 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:30:42.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:30:42.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:30:42.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:30:42.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:30:42.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:30:42.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-16T15:30:42.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-16T15:30:42.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:30:42.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:30:42.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:30:42.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:30:42.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:30:42.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:30:42.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2024-09-16T15:30:42.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:30:42.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=39de9123-29d9-419f-a540-cf625a7bdaf8 2024-09-16T15:30:42.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 39de9123-29d9-419f-a540-cf625a7bdaf8' 2024-09-16T15:30:42.519 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 39de9123-29d9-419f-a540-cf625a7bdaf8 2024-09-16T15:30:42.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:30:42.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAiT+hm7ItJIBAAxx6xcNMp6QtqMTPfWrD5RA== 2024-09-16T15:30:42.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAiT+hm7ItJIBAAxx6xcNMp6QtqMTPfWrD5RA=="}' 2024-09-16T15:30:42.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 39de9123-29d9-419f-a540-cf625a7bdaf8 -i td/ec-error-rollforward/1/new.json 2024-09-16T15:30:43.055 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:30:43.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2024-09-16T15:30:43.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAiT+hm7ItJIBAAxx6xcNMp6QtqMTPfWrD5RA== --osd-uuid 39de9123-29d9-419f-a540-cf625a7bdaf8 2024-09-16T15:30:43.098 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:43.099+0000 7f0017f9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:43.103 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:43.107+0000 7f0017f9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:43.107+0000 7f0017f9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:43.107+0000 7f0017f9b7c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-09-16T15:30:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:43.107+0000 7f0017f9b7c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-09-16T15:30:43.105 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:43.107+0000 7f0017f9b7c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-09-16T15:30:43.144 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:43.143+0000 7f0017f9b7c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-09-16T15:30:49.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-09-16T15:30:49.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:30:49.837 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:30:49.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:30:49.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:30:50.415 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:30:50.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:30:50.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:30:50.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:30:50.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:30:50.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:30:50.445 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:50.446+0000 7fb9740467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:50.477 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:50.478+0000 7fb9740467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:50.478 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:50.478+0000 7fb9740467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:30:50.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:30:50.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:30:50.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:30:50.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:30:50.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:30:50.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:50.856 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:30:50.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:30:50.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:50.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:30:51.049 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:51.050+0000 7fb9740467c0 -1 Falling back to public interface 2024-09-16T15:30:51.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:52.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:52.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:52.270 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:30:52.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:30:52.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:52.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:30:52.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:53.653 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:30:53.654+0000 7fb9740467c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:30:53.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:53.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:53.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:30:53.747 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:30:53.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:53.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:30:54.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:55.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:55.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:55.360 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:30:55.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:30:55.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:55.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:30:55.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:56.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:56.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:56.843 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:30:56.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:30:56.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:56.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:30:57.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:58.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:58.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:58.276 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:30:58.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:30:58.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:58.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:30:58.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:30:59.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:30:59.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:30:59.749 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:30:59.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:30:59.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:30:59.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:01.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:01.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:01.324 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:31:01.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:31:01.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:01.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:01.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:02.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:02.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:02.793 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:31:02.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:31:02.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:02.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:03.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:04.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:04.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:04.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:31:04.230 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:31:04.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:04.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:04.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:05.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:05.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:05.753 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:31:05.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:31:05.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:05.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:06.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:07.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:07.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:07.502 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:31:07.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:31:07.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:07.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:08.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:09.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:09.129 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:31:09.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:31:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:10.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:11.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:11.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:11.417 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:31:11.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:31:11.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:11.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:12.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:13.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:13.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:13.181 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:31:13.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:31:13.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:13.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:14.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:15.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:15.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:15.077 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:31:15.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:31:15.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:15.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:15.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:16.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:16.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:16.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:31:16.888 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:31:16.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:16.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:17.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:18.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:18.692 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:31:18.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:18.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:31:18.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:18.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:31:19.196 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2206171321,v1:127.0.0.1:6811/2206171321] [v2:127.0.0.1:6812/2206171321,v1:127.0.0.1:6813/2206171321] exists,up 39de9123-29d9-419f-a540-cf625a7bdaf8 2024-09-16T15:31:19.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:31:19.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:31:19.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:31:19.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:31:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-16T15:31:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:31:19.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:31:19.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:31:19.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:31:19.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:31:19.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:31:19.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2024-09-16T15:31:19.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:31:19.203 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 5a2f5a6d-1f7f-4a2f-9c9d-790dcbe672ed 2024-09-16T15:31:19.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5a2f5a6d-1f7f-4a2f-9c9d-790dcbe672ed 2024-09-16T15:31:19.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5a2f5a6d-1f7f-4a2f-9c9d-790dcbe672ed' 2024-09-16T15:31:19.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:31:19.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHT+hmesh8DRAAhKqYnWAR3jPVLNL5q758sA== 2024-09-16T15:31:19.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHT+hmesh8DRAAhKqYnWAR3jPVLNL5q758sA=="}' 2024-09-16T15:31:19.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5a2f5a6d-1f7f-4a2f-9c9d-790dcbe672ed -i td/ec-error-rollforward/2/new.json 2024-09-16T15:31:19.774 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:31:19.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2024-09-16T15:31:19.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHT+hmesh8DRAAhKqYnWAR3jPVLNL5q758sA== --osd-uuid 5a2f5a6d-1f7f-4a2f-9c9d-790dcbe672ed 2024-09-16T15:31:19.825 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:19.826+0000 7ff65d58f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:31:19.827 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:19.830+0000 7ff65d58f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:31:19.828 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:19.830+0000 7ff65d58f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:31:19.829 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:19.830+0000 7ff65d58f7c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-09-16T15:31:19.829 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:19.830+0000 7ff65d58f7c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-09-16T15:31:19.829 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:19.830+0000 7ff65d58f7c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-09-16T15:31:19.929 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:19.930+0000 7ff65d58f7c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-09-16T15:31:26.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-09-16T15:31:26.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:31:26.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T15:31:26.028 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T15:31:26.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:31:26.603 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:31:26.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T15:31:26.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:31:26.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:31:26.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:31:26.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:31:26.635 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:26.634+0000 7f3a918c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:31:26.652 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:26.654+0000 7f3a918c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:31:26.653 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:26.654+0000 7f3a918c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:31:27.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T15:31:27.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:31:27.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:31:27.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:31:27.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:31:27.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:27.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:31:27.028 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:31:27.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:27.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:27.229 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:27.230+0000 7f3a918c97c0 -1 Falling back to public interface 2024-09-16T15:31:27.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:28.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:28.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:28.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:31:28.462 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:31:28.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:28.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:28.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:29.645 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:29.646+0000 7f3a918c97c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:31:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:29.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:29.881 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:31:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:31:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:29.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:30.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:31.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:31.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:31.418 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:31:31.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:31:31.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:31.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:31.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:32.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:32.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:32.866 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:31:32.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:31:32.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:32.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:33.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:34.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:34.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:34.317 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:31:34.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:31:34.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:34.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:34.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:35.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:35.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:35.771 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:31:35.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:31:35.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:35.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:36.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:37.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:37.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:37.407 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:31:37.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:31:37.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:37.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:37.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:38.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:38.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:38.892 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:31:38.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:31:38.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:38.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:39.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:40.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:40.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:40.341 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:31:40.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:31:40.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:40.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:40.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:41.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:41.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:41.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:31:41.873 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:31:41.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:41.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:42.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:43.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:31:43.411 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:31:43.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:43.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:44.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:45.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:45.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:45.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:31:45.056 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:31:45.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:45.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:45.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:46.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:46.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:46.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:31:46.972 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:31:46.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:46.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:47.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:48.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:48.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:48.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:31:48.558 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:31:48.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:48.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:49.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:50.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:50.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:50.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:31:50.400 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:31:50.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:50.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:51.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:52.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:52.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:52.163 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:31:52.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:31:52.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:52.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:53.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:54.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:54.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:54.013 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:31:54.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:31:54.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:54.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:54.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:31:55.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:31:55.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:31:55.843 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T15:31:55.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T15:31:55.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:31:55.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:31:56.310 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2476856565,v1:127.0.0.1:6819/2476856565] [v2:127.0.0.1:6820/2476856565,v1:127.0.0.1:6821/2476856565] exists,up 5a2f5a6d-1f7f-4a2f-9c9d-790dcbe672ed 2024-09-16T15:31:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:31:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:31:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:31:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-09-16T15:31:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-16T15:31:56.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:31:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T15:31:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:31:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2024-09-16T15:31:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-16T15:31:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:31:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:31:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:31:56.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-16T15:31:56.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-16T15:31:56.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:31:56.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:31:56.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-16T15:31:56.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:31:56.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:31:56.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:31:56.318 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:31:56.318 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:31:56.318 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:31:56.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:31:56.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:31:56.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:31:56.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:31:56.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:31:56.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-16T15:31:56.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-16T15:31:56.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:31:56.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:31:56.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:31:56.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:31:56.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:31:56.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:31:56.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2024-09-16T15:31:56.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:31:56.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6c7795d3-0cc6-464e-bfd4-e72ff503de07 2024-09-16T15:31:56.322 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 6c7795d3-0cc6-464e-bfd4-e72ff503de07 2024-09-16T15:31:56.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 6c7795d3-0cc6-464e-bfd4-e72ff503de07' 2024-09-16T15:31:56.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:31:56.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBsT+hmjMM6FBAAoOkq6WOMndna1Ihm0Netrg== 2024-09-16T15:31:56.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBsT+hmjMM6FBAAoOkq6WOMndna1Ihm0Netrg=="}' 2024-09-16T15:31:56.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6c7795d3-0cc6-464e-bfd4-e72ff503de07 -i td/ec-error-rollforward/3/new.json 2024-09-16T15:31:56.930 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:31:56.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2024-09-16T15:31:56.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsT+hmjMM6FBAAoOkq6WOMndna1Ihm0Netrg== --osd-uuid 6c7795d3-0cc6-464e-bfd4-e72ff503de07 2024-09-16T15:31:56.983 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:56.985+0000 7f0e69fe37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:31:56.986 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:56.985+0000 7f0e69fe37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:31:56.987 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:56.989+0000 7f0e69fe37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:31:56.988 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:56.989+0000 7f0e69fe37c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-09-16T15:31:56.988 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:56.989+0000 7f0e69fe37c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-09-16T15:31:56.988 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:56.989+0000 7f0e69fe37c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-09-16T15:31:57.069 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:31:57.069+0000 7f0e69fe37c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-09-16T15:32:03.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-09-16T15:32:03.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:32:03.549 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T15:32:03.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T15:32:03.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:32:04.118 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T15:32:04.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T15:32:04.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:32:04.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:32:04.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:32:04.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:32:04.149 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:04.149+0000 7fea5ecc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:04.159 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:04.161+0000 7fea5ecc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:04.160 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:04.161+0000 7fea5ecc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:04.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T15:32:04.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:32:04.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T15:32:04.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:32:04.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:32:04.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:04.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:32:04.536 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:32:04.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:04.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:04.709 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:04.709+0000 7fea5ecc07c0 -1 Falling back to public interface 2024-09-16T15:32:04.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:05.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:05.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:05.982 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:32:05.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:32:05.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:05.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:06.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:07.279 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:07.281+0000 7fea5ecc07c0 -1 osd.3 0 log_to_monitors true 2024-09-16T15:32:07.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:07.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:07.423 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:32:07.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:32:07.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:07.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:08.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:09.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:09.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:09.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:32:09.233 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:32:09.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:09.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:09.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:10.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:10.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:10.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:32:10.661 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:32:10.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:10.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:11.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:12.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:12.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:12.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:32:12.157 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:32:12.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:12.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:12.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:13.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:13.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:13.713 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:32:13.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:32:13.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:13.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:14.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:15.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:15.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:15.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:32:15.288 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:32:15.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:15.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:15.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:16.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:16.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:16.719 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:32:16.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:32:16.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:16.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:17.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:18.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:18.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:18.328 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:32:18.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:32:18.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:18.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:18.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:19.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:19.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:19.944 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:32:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:32:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:20.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:21.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:21.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:21.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:32:21.378 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:32:21.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:21.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:21.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:22.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:22.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:22.829 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:32:22.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:32:22.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:22.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:23.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:24.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:24.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:24.634 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:32:24.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:32:24.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:24.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:25.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:26.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:26.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:26.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:32:26.466 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:32:26.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:26.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:27.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:28.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:28.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:28.610 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:32:28.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:32:28.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:28.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:29.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:30.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:30.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:30.758 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:32:30.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:32:30.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:30.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:31.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:32:32.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:32:32.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:32.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:32:32.594 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:32:32.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:32.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:33.074 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4219941480,v1:127.0.0.1:6827/4219941480] [v2:127.0.0.1:6828/4219941480,v1:127.0.0.1:6829/4219941480] exists,up 6c7795d3-0cc6-464e-bfd4-e72ff503de07 2024-09-16T15:32:33.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:32:33.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:32:33.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:32:33.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-16T15:32:33.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-09-16T15:32:34.520 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ec' already exists 2024-09-16T15:32:34.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-09-16T15:32:36.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-09-16T15:32:36.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 88585 2024-09-16T15:32:36.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=91320 2024-09-16T15:32:36.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-09-16T15:32:36.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-09-16T15:32:37.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 91341' 2024-09-16T15:32:37.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-09-16T15:32:37.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 91342' 2024-09-16T15:32:37.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-09-16T15:32:37.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 91343' 2024-09-16T15:32:37.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-09-16T15:32:37.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-09-16T15:32:38.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-09-16T15:32:38.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:32:38.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:32:38.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:32:38.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:32:38.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:32:38.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:32:38.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 91320 91341 91342 91343 2024-09-16T15:32:38.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-09-16T15:32:38.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-09-16T15:32:38.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-16T15:32:38.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:32:38.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:32:38.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:32:38.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-09-16T15:32:38.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-16T15:32:38.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:32:38.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:32:38.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:32:38.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-16T15:32:38.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-16T15:32:38.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:32:38.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:32:38.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-16T15:32:38.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:32:38.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:32:38.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:32:38.166 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:32:38.166 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:38.166 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:32:38.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:32:38.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:32:38.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:32:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-16T15:32:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-16T15:32:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:32:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:32:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:32:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:32:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:32:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:32:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-09-16T15:32:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:32:38.168 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:32:38.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:32:38.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-09-16T15:32:38.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:32:38.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:32:38.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:32:38.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:32:38.191 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:38.193+0000 7f0eff6b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:38.194 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:38.193+0000 7f0eff6b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:38.195 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:38.197+0000 7f0eff6b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:32:38.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:32:38.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:32:38.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:32:38.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:32:38.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:38.576 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:32:38.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:32:38.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:38.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:32:38.749 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:38.749+0000 7f0eff6b77c0 -1 Falling back to public interface 2024-09-16T15:32:39.209 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/293590132,v1:127.0.0.1:6803/293590132] [v2:127.0.0.1:6804/293590132,v1:127.0.0.1:6805/293590132] exists,up 91501509-0cc7-468c-8073-f1575b15d39c 2024-09-16T15:32:39.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:32:39.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:32:39.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:32:39.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-09-16T15:32:39.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-16T15:32:39.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:32:39.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:32:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:32:39.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:32:39.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:32:39.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-09-16T15:32:39.215 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:32:39.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:32:39.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:32:39.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-09-16T15:32:39.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:32:39.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:32:39.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:32:39.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:32:39.236 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:39.237+0000 7fa7a10787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:39.239 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:39.241+0000 7fa7a10787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:39.240 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:39.241+0000 7fa7a10787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:39.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T15:32:39.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:32:39.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:32:39.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:32:39.640 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:32:39.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:32:39.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:39.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:32:39.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:39.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:32:39.812 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:39.813+0000 7fa7a10787c0 -1 Falling back to public interface 2024-09-16T15:32:40.116 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 22 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2206171321,v1:127.0.0.1:6811/2206171321] [v2:127.0.0.1:6812/2206171321,v1:127.0.0.1:6813/2206171321] exists,up 39de9123-29d9-419f-a540-cf625a7bdaf8 2024-09-16T15:32:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:32:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:32:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:32:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-09-16T15:32:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-16T15:32:40.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:32:40.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T15:32:40.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:32:40.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-09-16T15:32:40.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-16T15:32:40.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:32:40.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:32:40.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:32:40.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-16T15:32:40.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:32:40.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:32:40.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-16T15:32:40.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-16T15:32:40.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:32:40.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:32:40.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:32:40.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:32:40.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:32:40.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:32:40.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-09-16T15:32:40.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T15:32:40.123 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:32:40.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:32:40.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-09-16T15:32:40.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T15:32:40.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:32:40.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:32:40.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:32:40.144 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:40.145+0000 7f1f5622f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:40.146 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:40.145+0000 7f1f5622f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:40.147 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:40.149+0000 7f1f5622f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:40.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T15:32:40.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:32:40.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:32:40.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:32:40.572 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:32:40.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:32:40.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:40.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:32:40.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:40.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:32:40.720 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:40.721+0000 7f1f5622f7c0 -1 Falling back to public interface 2024-09-16T15:32:41.002 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2476856565,v1:127.0.0.1:6819/2476856565] [v2:127.0.0.1:6820/2476856565,v1:127.0.0.1:6821/2476856565] exists,up 5a2f5a6d-1f7f-4a2f-9c9d-790dcbe672ed 2024-09-16T15:32:41.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:32:41.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:32:41.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:32:41.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-09-16T15:32:41.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-16T15:32:41.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:32:41.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-16T15:32:41.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:32:41.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:32:41.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:32:41.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:32:41.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:32:41.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-09-16T15:32:41.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-16T15:32:41.009 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T15:32:41.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=835b124c-d964-4dcc-a007-76cbae03e2ae --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84027/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:32:41.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-09-16T15:32:41.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-16T15:32:41.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:32:41.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:32:41.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:32:41.029 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:41.028+0000 7f72d840e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:41.031 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:41.032+0000 7f72d840e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:41.032 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:41.032+0000 7f72d840e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:41.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-16T15:32:41.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:32:41.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T15:32:41.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:32:41.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:32:41.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:32:41.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:32:41.519 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:32:41.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:32:41.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:32:41.602 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:41.600+0000 7f72d840e7c0 -1 Falling back to public interface 2024-09-16T15:32:42.039 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4219941480,v1:127.0.0.1:6827/4219941480] [v2:127.0.0.1:6828/4219941480,v1:127.0.0.1:6829/4219941480] exists,up 6c7795d3-0cc6-464e-bfd4-e72ff503de07 2024-09-16T15:32:42.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:32:42.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:32:42.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:32:42.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-09-16T15:32:42.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:32:42.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:32:42.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:32:42.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:32:42.042 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:32:42.042 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:32:42.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:32:42.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:32:42.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:32:42.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:32:42.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:32:42.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:32:42.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:32:42.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:32:42.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:32:42.411 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:42.412+0000 7f0eff6b77c0 -1 osd.0 24 log_to_monitors true 2024-09-16T15:32:42.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:32:42.631 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:32:42.631 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T15:32:42.631 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T15:32:42.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:32:42.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:32:42.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:32:43.627 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:43.628+0000 7fa7a10787c0 -1 osd.1 24 log_to_monitors true 2024-09-16T15:32:45.086 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:45.084+0000 7f1f5622f7c0 -1 osd.2 23 log_to_monitors true 2024-09-16T15:32:45.772 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:45.772+0000 7f72d840e7c0 -1 osd.3 24 log_to_monitors true 2024-09-16T15:32:47.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=115964116995 2024-09-16T15:32:47.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 115964116995 2024-09-16T15:32:47.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-115964116995' 2024-09-16T15:32:47.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:32:47.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:32:48.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018883 2024-09-16T15:32:48.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018883 2024-09-16T15:32:48.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-115964116995 1-128849018883' 2024-09-16T15:32:48.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:32:48.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:32:48.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=146028888067 2024-09-16T15:32:48.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 146028888067 2024-09-16T15:32:48.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-115964116995 1-128849018883 2-146028888067' 2024-09-16T15:32:48.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:32:48.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T15:32:48.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=150323855363 2024-09-16T15:32:48.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 150323855363 2024-09-16T15:32:48.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-115964116995 1-128849018883 2-146028888067 3-150323855363' 2024-09-16T15:32:48.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:32:48.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-115964116995 2024-09-16T15:32:48.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:32:48.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:32:48.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-115964116995 2024-09-16T15:32:48.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:32:48.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=115964116995 2024-09-16T15:32:48.403 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 115964116995 2024-09-16T15:32:48.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 115964116995' 2024-09-16T15:32:48.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:32:48.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 115964116995 2024-09-16T15:32:48.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:32:49.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:32:49.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:32:50.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 115964116996 -lt 115964116995 2024-09-16T15:32:50.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:32:50.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-128849018883 2024-09-16T15:32:50.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:32:50.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:32:50.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-128849018883 2024-09-16T15:32:50.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:32:50.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018883 2024-09-16T15:32:50.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 128849018883' 2024-09-16T15:32:50.276 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 128849018883 2024-09-16T15:32:50.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:32:50.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018884 -lt 128849018883 2024-09-16T15:32:50.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:32:50.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-146028888067 2024-09-16T15:32:50.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:32:50.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:32:50.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-146028888067 2024-09-16T15:32:50.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:32:50.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=146028888067 2024-09-16T15:32:50.741 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 146028888067 2024-09-16T15:32:50.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2024-09-16T15:32:50.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:32:51.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 146028888067 -lt 146028888067 2024-09-16T15:32:51.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:32:51.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:32:51.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-150323855363 2024-09-16T15:32:51.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T15:32:51.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-150323855363 2024-09-16T15:32:51.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:32:51.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=150323855363 2024-09-16T15:32:51.225 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 150323855363 2024-09-16T15:32:51.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2024-09-16T15:32:51.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T15:32:51.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 150323855363 -lt 150323855363 2024-09-16T15:32:51.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:32:51.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:32:51.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:32:52.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T15:32:52.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:32:52.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:32:52.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:32:52.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:32:52.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:32:52.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:32:52.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:32:52.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T15:32:52.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:32:52.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:32:52.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:32:53.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T15:32:53.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:32:53.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:32:53.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-09-16T15:32:53.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-16T15:32:53.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:32:53.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-16T15:32:53.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:32:53.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:32:53.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:32:53.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:32:53.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:32:53.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:32:53.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:32:53.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:32:53.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:32:53.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:32:53.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:32:53.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:32:53.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:32:53.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:32:53.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:32:53.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:32:53.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:32:53.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:32:53.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-16T15:32:53.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:32:53.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:53.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:32:53.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.84027 2024-09-16T15:32:53.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:32:53.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:32:53.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T15:32:53.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/ec-error-rollforward 0 2024-09-16T15:32:53.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-16T15:32:53.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T15:32:53.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-16T15:32:53.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:32:53.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:32:53.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:32:53.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:32:53.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:32:53.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:32:53.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:32:53.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:32:53.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:32:53.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:32:53.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:32:53.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:32:53.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:32:53.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:32:53.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:32:53.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:32:53.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:32:53.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T15:32:53.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-16T15:32:53.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:32:53.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:53.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84027 2024-09-16T15:32:53.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.84027 2024-09-16T15:32:53.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:32:53.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:32:53.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T15:32:53.304 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T15:32:53.305 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T15:32:53.355 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-09-16T15:32:53.355 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-09-16T15:32:53.407 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T15:32:53.413 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T15:32:53.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-bench 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-09-16T15:32:53.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-09-16T15:32:53.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-09-16T15:32:53.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-09-16T15:32:53.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=7cbe249d-cd4e-4106-9a8c-b4b2ff1aeceb --auth-supported=none ' 2024-09-16T15:32:53.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-09-16T15:32:53.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-16T15:32:53.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T15:32:53.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-09-16T15:32:53.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-09-16T15:32:53.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-09-16T15:32:53.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-09-16T15:32:53.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2024-09-16T15:32:53.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2024-09-16T15:32:53.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-16T15:32:53.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:32:53.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-16T15:32:53.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:32:53.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:32:53.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:32:53.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:32:53.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:32:53.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:32:53.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:32:53.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:32:53.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:32:53.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:32:53.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:32:53.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:32:53.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:32:53.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:32:53.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:32:53.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:32:53.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:32:53.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:32:53.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-16T15:32:53.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:32:53.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:53.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93848 2024-09-16T15:32:53.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93848 2024-09-16T15:32:53.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:32:53.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:32:53.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2024-09-16T15:32:53.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:32:53.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:53.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93848 2024-09-16T15:32:53.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.93848 2024-09-16T15:32:53.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:32:53.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T15:32:53.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T15:32:53.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:32:53.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-09-16T15:32:53.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-09-16T15:32:53.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-09-16T15:32:53.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-09-16T15:32:53.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2024-09-16T15:32:53.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:32:53.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:32:53.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:32:53.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2024-09-16T15:32:53.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-09-16T15:32:53.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:32:53.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:32:53.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:32:53.847 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:32:53.848 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:53.848 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93848 2024-09-16T15:32:53.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93848/$cluster-$name.asok' 2024-09-16T15:32:53.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.93848/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:32:54.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:32:54.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:32:54.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:32:54.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:32:54.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:32:54.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:32:54.060 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:32:54.060 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:32:54.060 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:32:54.060 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:32:54.060 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:54.060 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93848 2024-09-16T15:32:54.060 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.93848/ceph-mon.a.asok 2024-09-16T15:32:54.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:32:54.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.93848/ceph-mon.a.asok config get fsid 2024-09-16T15:32:54.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:32:54.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:32:54.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:32:54.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:32:54.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:32:54.177 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:32:54.177 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:32:54.177 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:32:54.177 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:32:54.177 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:54.177 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93848 2024-09-16T15:32:54.178 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.93848/ceph-mon.a.asok 2024-09-16T15:32:54.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:32:54.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.93848/ceph-mon.a.asok config get mon_host 2024-09-16T15:32:54.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-09-16T15:32:54.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2024-09-16T15:32:54.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:32:54.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:32:54.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:32:54.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2024-09-16T15:32:54.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:32:55.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:32:55.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:32:55.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:32:55.190 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:32:55.190 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:55.190 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93848 2024-09-16T15:32:55.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93848/$cluster-$name.asok' 2024-09-16T15:32:55.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:32:55.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.93848/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:32:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-09-16T15:32:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2024-09-16T15:32:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:32:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:32:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:32:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2024-09-16T15:32:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7cbe249d-cd4e-4106-9a8c-b4b2ff1aeceb --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:32:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93848 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93848/$cluster-$name.asok' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.93848/$cluster-$name.asok' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2024-09-16T15:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=db955683-f3f9-427c-ac0a-67eb7ef7fde0 2024-09-16T15:32:55.224 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 db955683-f3f9-427c-ac0a-67eb7ef7fde0 2024-09-16T15:32:55.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 db955683-f3f9-427c-ac0a-67eb7ef7fde0' 2024-09-16T15:32:55.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:32:55.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnT+hm3ER4DhAAolIL6H4KRjggKBNl64pQHQ== 2024-09-16T15:32:55.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnT+hm3ER4DhAAolIL6H4KRjggKBNl64pQHQ=="}' 2024-09-16T15:32:55.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new db955683-f3f9-427c-ac0a-67eb7ef7fde0 -i td/osd-bench/0/new.json 2024-09-16T15:32:55.573 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:32:55.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2024-09-16T15:32:55.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7cbe249d-cd4e-4106-9a8c-b4b2ff1aeceb --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.93848/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnT+hm3ER4DhAAolIL6H4KRjggKBNl64pQHQ== --osd-uuid db955683-f3f9-427c-ac0a-67eb7ef7fde0 2024-09-16T15:32:55.624 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:55.624+0000 7f599259a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:55.630 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:55.628+0000 7f599259a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:55.631 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:55.632+0000 7f599259a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:32:55.632 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:55.632+0000 7f599259a7c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-09-16T15:32:55.632 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:55.632+0000 7f599259a7c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-09-16T15:32:55.632 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:55.632+0000 7f599259a7c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-09-16T15:32:55.670 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:32:55.668+0000 7f599259a7c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-09-16T15:33:02.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2024-09-16T15:33:02.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:33:02.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:33:02.070 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:33:02.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:33:02.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:33:02.709 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:33:02.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7cbe249d-cd4e-4106-9a8c-b4b2ff1aeceb --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.93848/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:33:02.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:33:02.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:33:02.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:33:02.740 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:33:02.740+0000 7f77d39227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:33:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:33:02.768+0000 7f77d39227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:33:02.771 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:33:02.772+0000 7f77d39227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:33:03.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:33:03.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:33:03.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:33:03.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:33:03.145 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:33:03.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:33:03.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:03.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:33:03.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:03.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:03.356 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:33:03.356+0000 7f77d39227c0 -1 Falling back to public interface 2024-09-16T15:33:03.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:04.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:04.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:04.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:33:04.637 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:33:04.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:04.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:05.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:05.977 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:33:05.976+0000 7f77d39227c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:33:06.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:06.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:06.103 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:33:06.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:33:06.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:06.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:06.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:07.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:07.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:07.653 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:33:07.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:33:07.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:07.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:08.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:09.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:09.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:09.143 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:33:09.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:33:09.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:09.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:09.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:10.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:10.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:10.600 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:33:10.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:33:10.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:10.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:11.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:12.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:12.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:12.090 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:33:12.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:33:12.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:12.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:12.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:13.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:13.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:13.529 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:33:13.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:33:13.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:13.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:13.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:14.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:14.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:14.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:33:14.956 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:33:14.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:14.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:15.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:16.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:16.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:16.391 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:33:16.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:33:16.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:16.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:16.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:17.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:17.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:17.943 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:33:17.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:33:17.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:17.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:18.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:19.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:19.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:19.370 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:33:19.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:33:19.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:19.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:19.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:20.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:20.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:20.895 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:33:20.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:33:20.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:20.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:21.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:22.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:22.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:22.693 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:33:22.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:33:22.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:22.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:23.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:24.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:24.536 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:33:24.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:24.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:33:24.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:24.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:25.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:26.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:26.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:26.641 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:33:26.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:33:26.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:26.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:27.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:33:28.847 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:33:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:29.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:33:30.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:33:30.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:33:30.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:33:30.858 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:33:30.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:33:30.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:33:31.290 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/421197569,v1:127.0.0.1:6803/421197569] [v2:127.0.0.1:6804/421197569,v1:127.0.0.1:6805/421197569] exists,up db955683-f3f9-427c-ac0a-67eb7ef7fde0 2024-09-16T15:33:31.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:33:31.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:33:31.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:33:31.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-09-16T15:33:31.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-09-16T15:33:31.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-09-16T15:33:31.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-09-16T15:33:31.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-09-16T15:33:31.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-09-16T15:33:31.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-09-16T15:33:31.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-09-16T15:33:31.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-09-16T15:33:31.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-09-16T15:33:31.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-09-16T15:33:31.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-09-16T15:33:31.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-09-16T15:33:31.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/osd-bench 2024-09-16T15:33:31.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2024-09-16T15:33:31.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local expected=osd_bench_max_block_size 2024-09-16T15:33:31.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2024-09-16T15:33:31.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2024-09-16T15:33:31.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-09-16T15:33:31.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2024-09-16T15:33:31.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2024-09-16T15:33:31.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-09-16T15:33:31.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: return 0 2024-09-16T15:33:31.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-09-16T15:33:31.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-09-16T15:33:31.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-09-16T15:33:31.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/osd-bench 2024-09-16T15:33:31.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2024-09-16T15:33:31.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local expected=bench_small_size_max_iops 2024-09-16T15:33:31.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2024-09-16T15:33:31.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2024-09-16T15:33:31.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-09-16T15:33:31.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2024-09-16T15:33:31.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2024-09-16T15:33:31.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-09-16T15:33:31.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: return 0 2024-09-16T15:33:31.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-09-16T15:33:31.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-09-16T15:33:31.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-09-16T15:33:31.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/osd-bench 2024-09-16T15:33:31.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2024-09-16T15:33:31.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-09-16T15:33:31.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2024-09-16T15:33:31.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2024-09-16T15:33:31.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-09-16T15:33:31.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2024-09-16T15:33:31.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2024-09-16T15:33:31.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-09-16T15:33:31.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: return 0 2024-09-16T15:33:31.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-09-16T15:34:26.063 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:34:26.063 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_written": 1073741824, 2024-09-16T15:34:26.063 INFO:tasks.workunit.client.0.smithi039.stdout: "blocksize": 4194304, 2024-09-16T15:34:26.064 INFO:tasks.workunit.client.0.smithi039.stdout: "elapsed_sec": 25.577045477999999, 2024-09-16T15:34:26.064 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_per_sec": 41980682.441354498, 2024-09-16T15:34:26.064 INFO:tasks.workunit.client.0.smithi039.stdout: "iops": 10.008974657381653 2024-09-16T15:34:26.064 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:34:26.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-09-16T15:34:26.593 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:34:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_written": 10, 2024-09-16T15:34:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "blocksize": 14456, 2024-09-16T15:34:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "elapsed_sec": 0.033479940999999999, 2024-09-16T15:34:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_per_sec": 298.6863089155384, 2024-09-16T15:34:26.594 INFO:tasks.workunit.client.0.smithi039.stdout: "iops": 0.020661753522104205 2024-09-16T15:34:26.594 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:34:26.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-09-16T15:34:26.911 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T15:34:26.911 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_written": 1, 2024-09-16T15:34:26.911 INFO:tasks.workunit.client.0.smithi039.stdout: "blocksize": 14456, 2024-09-16T15:34:26.911 INFO:tasks.workunit.client.0.smithi039.stdout: "elapsed_sec": 0.039382970000000003, 2024-09-16T15:34:26.911 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_per_sec": 25.391685797185939, 2024-09-16T15:34:26.911 INFO:tasks.workunit.client.0.smithi039.stdout: "iops": 0.0017564807552010195 2024-09-16T15:34:26.911 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T15:34:26.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-09-16T15:34:26.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-16T15:34:26.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:34:26.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-16T15:34:26.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:34:26.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:34:26.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:34:26.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:34:26.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:34:27.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:34:27.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:34:27.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:34:27.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:34:27.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:34:27.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:34:27.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:34:27.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:34:27.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:34:27.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:34:27.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:34:27.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:34:27.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:34:27.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-16T15:34:27.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:34:27.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:34:27.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93848 2024-09-16T15:34:27.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93848 2024-09-16T15:34:27.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:34:27.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:34:27.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T15:34:27.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-bench 0 2024-09-16T15:34:27.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-16T15:34:27.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T15:34:27.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-16T15:34:27.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:34:27.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:34:27.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:34:27.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:34:27.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:34:27.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:34:27.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:34:27.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:34:27.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:34:27.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:34:27.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:34:27.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:34:27.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:34:27.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:34:27.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:34:27.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:34:27.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:34:27.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T15:34:27.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-16T15:34:27.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:34:27.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:34:27.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93848 2024-09-16T15:34:27.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93848 2024-09-16T15:34:27.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:34:27.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:34:27.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T15:34:27.100 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T15:34:27.100 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T15:34:27.157 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-09-16T15:34:27.157 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-09-16T15:34:27.210 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T15:34:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T15:34:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:34:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:34:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T15:34:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T15:34:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T15:34:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T15:34:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T15:34:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T15:34:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-bluefs-volume-ops 2024-09-16T15:34:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:34:27.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-09-16T15:34:27.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-09-16T15:34:27.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T15:34:27.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_bluestore2 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_bluestore_expand' 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:34:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:34:27.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:34:27.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:34:27.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:34:27.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:34:27.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:34:27.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:34:27.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:34:27.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:34:27.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:34:27.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:34:27.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:34:27.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:34:27.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:34:27.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-16T15:34:27.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:34:27.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:34:27.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:34:27.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96021 2024-09-16T15:34:27.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:34:27.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:34:27.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-16T15:34:27.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:34:27.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:34:27.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:34:27.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.96021 2024-09-16T15:34:27.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:34:27.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T15:34:27.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T15:34:27.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:34:27.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-16T15:34:27.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-09-16T15:34:27.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:34:27.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-09-16T15:34:27.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-09-16T15:34:27.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-09-16T15:34:27.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-09-16T15:34:27.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-09-16T15:34:27.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-09-16T15:34:27.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-09-16T15:34:27.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none ' 2024-09-16T15:34:27.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-16T15:34:27.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-09-16T15:34:27.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-16T15:34:27.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-16T15:34:27.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-09-16T15:34:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-09-16T15:34:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-16T15:34:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-09-16T15:34:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:34:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:34:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:34:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:34:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-16T15:34:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-16T15:34:27.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:34:27.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:34:27.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:34:27.647 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:34:27.647 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:34:27.647 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:34:27.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:34:27.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:34:27.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:34:27.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:34:27.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:34:27.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:34:27.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:34:27.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:34:27.861 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:34:27.861 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:34:27.861 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:34:27.861 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:34:27.861 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:34:27.861 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:34:27.861 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96021/ceph-mon.a.asok 2024-09-16T15:34:27.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:34:27.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96021/ceph-mon.a.asok config get fsid 2024-09-16T15:34:27.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:34:27.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:34:27.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:34:27.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:34:27.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:34:27.984 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:34:27.984 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:34:27.984 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:34:27.984 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:34:27.984 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:34:27.984 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:34:27.985 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96021/ceph-mon.a.asok 2024-09-16T15:34:27.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:34:27.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96021/ceph-mon.a.asok config get mon_host 2024-09-16T15:34:28.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-09-16T15:34:28.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:34:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:34:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:34:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:34:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-16T15:34:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:34:29.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:34:29.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:34:29.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:34:29.018 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:34:29.018 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:34:29.018 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:34:29.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:34:29.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:34:29.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:34:29.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-09-16T15:34:29.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:34:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:34:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:34:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:34:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:34:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:34:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:34:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:34:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-16T15:34:29.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:34:29.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ae054d5-b804-4b87-94f9-fbc5503745dd 2024-09-16T15:34:29.087 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 6ae054d5-b804-4b87-94f9-fbc5503745dd 2024-09-16T15:34:29.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6ae054d5-b804-4b87-94f9-fbc5503745dd' 2024-09-16T15:34:29.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:34:29.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFUOhmmVFaBhAAetxTEgqmF5YCRy/CWoB0Uw== 2024-09-16T15:34:29.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFUOhmmVFaBhAAetxTEgqmF5YCRy/CWoB0Uw=="}' 2024-09-16T15:34:29.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ae054d5-b804-4b87-94f9-fbc5503745dd -i td/osd-bluefs-volume-ops/0/new.json 2024-09-16T15:34:29.465 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:34:29.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-16T15:34:29.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFUOhmmVFaBhAAetxTEgqmF5YCRy/CWoB0Uw== --osd-uuid 6ae054d5-b804-4b87-94f9-fbc5503745dd 2024-09-16T15:34:29.514 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:29.511+0000 7f0cb41f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:34:29.516 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:29.515+0000 7f0cb41f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:34:29.517 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:29.515+0000 7f0cb41f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:34:29.518 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:29.515+0000 7f0cb41f67c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-16T15:34:29.518 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:29.515+0000 7f0cb41f67c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-16T15:34:29.518 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:29.515+0000 7f0cb41f67c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-16T15:34:29.596 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:29.595+0000 7f0cb41f67c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-16T15:34:39.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-16T15:34:39.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:34:39.822 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:34:39.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:34:39.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:34:40.390 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:34:40.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:34:40.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:34:40.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:34:40.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:34:40.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:34:40.423 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:40.427+0000 7fa7dac097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:34:40.445 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:40.447+0000 7fa7dac097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:34:40.446 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:40.447+0000 7fa7dac097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:34:40.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:34:40.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:34:40.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:34:40.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:34:40.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:34:40.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:40.841 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:34:40.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:34:40.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:40.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:41.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:41.497 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:41.499+0000 7fa7dac097c0 -1 Falling back to public interface 2024-09-16T15:34:42.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:42.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:42.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:34:42.275 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:34:42.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:42.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:42.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:43.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:43.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:43.741 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:34:43.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:34:43.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:43.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:44.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:45.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:45.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:45.182 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:34:45.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:34:45.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:45.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:45.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:46.401 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:34:46.403+0000 7fa7dac097c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:34:46.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:46.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:46.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:34:46.605 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:34:46.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:46.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:47.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:48.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:48.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:48.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:34:48.084 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:34:48.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:48.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:48.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:49.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:49.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:49.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:34:49.514 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:34:49.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:49.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:49.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:50.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:50.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:50.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:34:50.949 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:34:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:51.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:52.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:52.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:52.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:34:52.474 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:34:52.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:52.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:52.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:53.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:53.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:53.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:34:53.934 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:34:53.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:53.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:54.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:55.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:55.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:55.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:34:55.364 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:34:55.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:55.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:55.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:56.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:56.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:56.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:34:56.791 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:34:56.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:56.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:57.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:58.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:58.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:58.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:34:58.234 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:34:58.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:58.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:34:58.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:34:59.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:34:59.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:34:59.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:34:59.792 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:34:59.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:34:59.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:35:00.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:01.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:01.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:01.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:35:01.613 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:35:01.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:01.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:35:02.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:03.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:03.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:03.446 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:35:03.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:35:03.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:03.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:35:04.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:05.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:05.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:05.277 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:35:05.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:35:05.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:05.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:35:06.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:07.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:07.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:07.209 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:35:07.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:35:07.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:07.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:35:07.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:08.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:08.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:08.854 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T15:35:08.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T15:35:08.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:08.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:35:09.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:10.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:10.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:10.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T15:35:10.658 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-09-16T15:35:10.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:10.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:35:11.138 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/541813198,v1:127.0.0.1:6803/541813198] [v2:127.0.0.1:6804/541813198,v1:127.0.0.1:6805/541813198] exists,up 6ae054d5-b804-4b87-94f9-fbc5503745dd 2024-09-16T15:35:11.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:35:11.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:35:11.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:35:11.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-16T15:35:11.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=97482 2024-09-16T15:35:11.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-09-16T15:35:11.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:35:11.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:35:11.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:35:11.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:35:11.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:35:11.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:35:11.145 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:35:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:35:11.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:35:11.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-16T15:35:11.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:35:11.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb1b948c-057b-433b-8f34-8d78c69ce8c7 2024-09-16T15:35:11.147 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 eb1b948c-057b-433b-8f34-8d78c69ce8c7 2024-09-16T15:35:11.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 eb1b948c-057b-433b-8f34-8d78c69ce8c7' 2024-09-16T15:35:11.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:35:11.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvUOhmzcZDChAArGolZe6e5vEcJRSm7U9kxQ== 2024-09-16T15:35:11.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvUOhmzcZDChAArGolZe6e5vEcJRSm7U9kxQ=="}' 2024-09-16T15:35:11.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb1b948c-057b-433b-8f34-8d78c69ce8c7 -i td/osd-bluefs-volume-ops/1/new.json 2024-09-16T15:35:11.894 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:35:11.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-09-16T15:35:11.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvUOhmzcZDChAArGolZe6e5vEcJRSm7U9kxQ== --osd-uuid eb1b948c-057b-433b-8f34-8d78c69ce8c7 2024-09-16T15:35:11.953 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:11.954+0000 7fbe717b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:35:11.955 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:11.958+0000 7fbe717b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:35:11.957 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:11.958+0000 7fbe717b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:35:11.957 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:11.958+0000 7fbe717b87c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-09-16T15:35:11.957 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:11.958+0000 7fbe717b87c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-09-16T15:35:11.957 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:11.958+0000 7fbe717b87c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-09-16T15:35:12.033 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:12.034+0000 7fbe717b87c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-09-16T15:35:22.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-09-16T15:35:22.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:35:22.232 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:35:22.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:35:22.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:35:22.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:35:22.817 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:35:22.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:35:22.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:35:22.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:35:22.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:35:22.841 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:22.842+0000 7f41c1d947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:35:22.864 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:22.866+0000 7f41c1d947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:35:22.865 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:22.866+0000 7f41c1d947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:35:23.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:35:23.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:35:23.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:35:23.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:35:23.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:35:23.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:23.289 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:35:23.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:35:23.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:23.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:23.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:23.929 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:23.930+0000 7f41c1d947c0 -1 Falling back to public interface 2024-09-16T15:35:24.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:24.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:24.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:35:24.722 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:35:24.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:24.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:26.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:26.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:35:26.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:35:26.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:26.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:26.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:27.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:27.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:27.562 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:35:27.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:35:27.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:27.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:27.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:28.810 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:28.814+0000 7f41c1d947c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:35:28.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:28.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:28.996 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:35:28.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:35:28.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:28.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:29.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:30.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:30.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:30.584 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:35:30.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:35:30.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:30.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:31.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:32.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:32.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:32.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:35:32.107 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:35:32.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:32.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:32.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:33.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:33.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:33.647 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:35:33.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:35:33.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:33.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:34.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:35.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:35.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:35.075 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:35:35.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:35:35.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:35.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:35.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:36.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:36.515 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:35:36.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:36.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:35:36.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:36.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:36.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:37.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:37.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:37.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:35:37.941 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:35:37.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:37.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:38.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:39.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:39.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:39.472 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:35:39.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:35:39.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:39.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:40.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:41.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:41.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:41.049 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:35:41.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:35:41.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:41.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:41.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:42.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:42.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:42.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:35:42.475 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:35:42.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:42.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:43.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:44.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:44.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:44.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:35:44.319 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:35:44.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:44.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:44.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:45.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:45.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:45.831 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:35:45.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:35:45.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:45.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:46.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:47.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:47.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:47.261 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:35:47.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:35:47.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:47.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:48.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:49.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:49.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:49.364 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:35:49.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:35:49.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:49.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:50.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:51.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:51.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:51.213 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T15:35:51.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T15:35:51.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:51.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:51.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:52.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:52.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:52.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T15:35:52.993 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-09-16T15:35:52.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:52.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:53.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:35:54.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:35:54.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:35:54.544 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T15:35:54.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T15:35:54.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:35:54.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:35:55.036 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2408668665,v1:127.0.0.1:6811/2408668665] [v2:127.0.0.1:6812/2408668665,v1:127.0.0.1:6813/2408668665] exists,up eb1b948c-057b-433b-8f34-8d78c69ce8c7 2024-09-16T15:35:55.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:35:55.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:35:55.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:35:55.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-16T15:35:55.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=100028 2024-09-16T15:35:55.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-09-16T15:35:55.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:35:55.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:35:55.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:35:55.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:35:55.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:35:55.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:35:55.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:35:55.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:35:55.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-16T15:35:55.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:35:55.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d3a747ca-52f9-47a9-abe1-7d1cec101ba3 2024-09-16T15:35:55.045 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 d3a747ca-52f9-47a9-abe1-7d1cec101ba3 2024-09-16T15:35:55.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d3a747ca-52f9-47a9-abe1-7d1cec101ba3' 2024-09-16T15:35:55.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:35:55.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbUOhmx8D1AxAAbR5MolwMMzC1rtGV1GFv9Q== 2024-09-16T15:35:55.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbUOhmx8D1AxAAbR5MolwMMzC1rtGV1GFv9Q=="}' 2024-09-16T15:35:55.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d3a747ca-52f9-47a9-abe1-7d1cec101ba3 -i td/osd-bluefs-volume-ops/2/new.json 2024-09-16T15:35:55.621 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:35:55.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-09-16T15:35:55.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbUOhmx8D1AxAAbR5MolwMMzC1rtGV1GFv9Q== --osd-uuid d3a747ca-52f9-47a9-abe1-7d1cec101ba3 2024-09-16T15:35:55.665 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:55.665+0000 7f7a1d1887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:35:55.670 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:55.669+0000 7f7a1d1887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:35:55.671 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:55.673+0000 7f7a1d1887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:35:55.671 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:55.673+0000 7f7a1d1887c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-09-16T15:35:55.672 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:55.673+0000 7f7a1d1887c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-09-16T15:35:55.672 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:55.673+0000 7f7a1d1887c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-09-16T15:35:55.710 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:35:55.709+0000 7f7a1d1887c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-09-16T15:36:06.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-09-16T15:36:06.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:36:06.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T15:36:06.184 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T15:36:06.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:36:06.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T15:36:06.747 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:36:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:36:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:36:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:36:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:36:06.778 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:06.777+0000 7f8407dce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:36:06.793 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:06.793+0000 7f8407dce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:36:06.794 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:06.797+0000 7f8407dce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:36:07.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T15:36:07.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:36:07.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:36:07.161 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:36:07.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:36:07.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:36:07.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:07.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:36:07.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:07.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:07.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:07.849 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:07.849+0000 7f8407dce7c0 -1 Falling back to public interface 2024-09-16T15:36:08.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:08.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:08.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:36:08.619 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:36:08.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:08.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:09.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:36:10.050 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:36:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:10.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:10.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:11.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:11.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:11.612 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:36:11.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:36:11.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:11.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:12.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:12.780 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:12.781+0000 7f8407dce7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:36:13.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:13.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:13.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:36:13.052 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:36:13.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:13.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:13.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:14.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:14.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:14.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:36:14.740 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:36:14.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:14.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:15.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:16.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:16.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:16.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:36:16.184 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:36:16.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:16.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:16.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:17.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:17.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:17.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:36:17.895 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:36:17.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:17.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:18.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:19.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:19.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:19.447 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:36:19.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:36:19.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:19.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:20.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:21.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:21.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:21.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:36:21.134 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:36:21.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:21.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:21.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:22.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:22.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:22.705 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:36:22.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:36:22.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:22.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:23.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:24.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:24.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:24.139 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:36:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:36:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:24.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:24.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:25.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:25.663 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:36:25.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:25.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:36:25.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:25.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:26.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:27.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:27.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:27.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:36:27.133 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:36:27.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:27.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:27.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:28.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:28.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:28.900 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:36:28.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:36:28.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:28.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:29.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:30.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:30.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:30.587 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:36:30.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:36:30.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:30.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:31.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:32.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:32.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:32.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:36:32.670 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:36:32.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:32.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:33.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:34.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:34.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:34.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:36:34.611 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:36:34.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:34.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:35.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:36.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:36.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:36.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T15:36:36.775 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T15:36:36.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:36.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:37.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:38.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:38.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:38.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T15:36:38.635 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-09-16T15:36:38.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:38.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:36:39.110 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3678077548,v1:127.0.0.1:6819/3678077548] [v2:127.0.0.1:6820/3678077548,v1:127.0.0.1:6821/3678077548] exists,up d3a747ca-52f9-47a9-abe1-7d1cec101ba3 2024-09-16T15:36:39.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:36:39.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:36:39.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:36:39.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-16T15:36:39.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=102594 2024-09-16T15:36:39.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-09-16T15:36:39.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:36:39.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:36:39.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T15:36:39.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:36:39.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-16T15:36:39.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:36:39.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:36:39.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:36:39.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:36:39.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-16T15:36:39.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-16T15:36:39.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:36:39.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:36:39.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:36:39.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:36:39.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:36:39.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:36:39.116 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:36:39.116 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:36:39.116 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:36:39.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:36:39.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:36:39.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:36:39.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:36:39.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:36:39.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:36:39.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:36:39.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:36:39.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:36:39.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:36:39.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:36:39.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:36:39.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:36:39.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-16T15:36:39.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:36:39.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8a48c2fe-87da-456e-960f-2bc0d19675f4 2024-09-16T15:36:39.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 8a48c2fe-87da-456e-960f-2bc0d19675f4' 2024-09-16T15:36:39.118 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 8a48c2fe-87da-456e-960f-2bc0d19675f4 2024-09-16T15:36:39.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:36:39.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHUOhmTmgrCBAAKfojKf8p2HTa7jrT09NACg== 2024-09-16T15:36:39.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHUOhmTmgrCBAAKfojKf8p2HTa7jrT09NACg=="}' 2024-09-16T15:36:39.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8a48c2fe-87da-456e-960f-2bc0d19675f4 -i td/osd-bluefs-volume-ops/3/new.json 2024-09-16T15:36:39.696 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:36:39.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-09-16T15:36:39.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHUOhmTmgrCBAAKfojKf8p2HTa7jrT09NACg== --osd-uuid 8a48c2fe-87da-456e-960f-2bc0d19675f4 2024-09-16T15:36:39.747 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:39.748+0000 7fcd051997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:36:39.749 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:39.748+0000 7fcd051997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:36:39.750 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:39.748+0000 7fcd051997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:36:39.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:39.752+0000 7fcd051997c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-09-16T15:36:39.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:39.752+0000 7fcd051997c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-09-16T15:36:39.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:39.752+0000 7fcd051997c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-09-16T15:36:39.865 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:39.864+0000 7fcd051997c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-09-16T15:36:50.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-09-16T15:36:50.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:36:50.274 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T15:36:50.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T15:36:50.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:36:50.829 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T15:36:50.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T15:36:50.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:36:50.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:36:50.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:36:50.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:36:50.861 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:50.860+0000 7f43e847f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:36:50.863 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:50.864+0000 7f43e847f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:36:50.865 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:50.864+0000 7f43e847f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:36:51.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T15:36:51.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:36:51.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T15:36:51.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:36:51.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:36:51.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:51.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:36:51.278 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:36:51.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:51.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:36:51.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:51.901 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:51.900+0000 7f43e847f7c0 -1 Falling back to public interface 2024-09-16T15:36:52.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:52.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:52.744 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:36:52.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:36:52.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:52.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:36:53.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:54.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:54.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:54.156 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:36:54.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:36:54.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:54.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:36:54.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:55.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:55.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:55.582 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:36:55.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:36:55.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:55.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:36:56.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:56.722 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:36:56.720+0000 7f43e847f7c0 -1 osd.3 0 log_to_monitors true 2024-09-16T15:36:57.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:57.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:57.012 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:36:57.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:36:57.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:57.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:36:57.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:36:58.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:36:58.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:36:58.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:36:58.681 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:36:58.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:36:58.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:36:59.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:00.152 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:37:00.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:00.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:00.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:37:00.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:00.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:00.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:01.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:01.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:01.611 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:37:01.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:37:01.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:01.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:02.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:03.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:03.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:03.101 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:37:03.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:37:03.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:03.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:03.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:04.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:04.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:04.665 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:37:04.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:37:04.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:04.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:05.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:06.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:06.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:06.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:37:06.103 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:37:06.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:06.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:06.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:07.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:07.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:07.566 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:37:07.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:37:07.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:07.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:08.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:09.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:09.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:09.138 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:37:09.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:37:09.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:09.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:09.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:10.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:10.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:10.622 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:37:10.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:37:10.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:10.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:11.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:12.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:12.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:12.343 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:37:12.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:37:12.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:12.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:13.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:14.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:14.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:14.109 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:37:14.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:37:14.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:14.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:15.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:16.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:16.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:16.343 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:37:16.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:37:16.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:16.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:17.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:18.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:18.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:18.255 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:37:18.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:37:18.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:18.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:19.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:20.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:20.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:20.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T15:37:20.366 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T15:37:20.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:20.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:21.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:37:22.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:37:22.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:37:22.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T15:37:22.174 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-09-16T15:37:22.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:37:22.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:37:22.597 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2151096827,v1:127.0.0.1:6827/2151096827] [v2:127.0.0.1:6828/2151096827,v1:127.0.0.1:6829/2151096827] exists,up 8a48c2fe-87da-456e-960f-2bc0d19675f4 2024-09-16T15:37:22.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:37:22.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:37:22.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:37:22.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-16T15:37:22.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=105136 2024-09-16T15:37:22.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-09-16T15:37:27.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-09-16T15:37:27.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-16T15:37:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T15:37:28.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:37:29.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-16T15:37:29.306 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-16T15:37:29.307 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-16T15:37:29.307 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_106492 2024-09-16T15:37:29.307 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:37:29.307 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-16T15:37:30.307 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 48 32 0.124986 0.125 0.297608 0.425492 2024-09-16T15:37:31.307 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 92 76 0.148417 0.171875 0.326355 0.363805 2024-09-16T15:37:32.307 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 140 124 0.161435 0.1875 0.348989 0.364298 2024-09-16T15:37:33.307 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 193 177 0.172827 0.207031 0.300121 0.347404 2024-09-16T15:37:34.308 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 249 233 0.182004 0.21875 0.308683 0.335988 2024-09-16T15:37:35.308 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 301 285 0.18552 0.203125 0.246618 0.330238 2024-09-16T15:37:36.308 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 351 335 0.186916 0.195312 0.314246 0.325425 2024-09-16T15:37:37.308 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 400 384 0.187475 0.191406 0.329005 0.32409 2024-09-16T15:37:38.308 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 456 440 0.190947 0.21875 0.326952 0.320686 2024-09-16T15:37:39.308 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 509 493 0.192553 0.207031 0.246616 0.320996 2024-09-16T15:37:40.308 INFO:tasks.workunit.client.0.smithi039.stdout: 11 16 557 541 0.192091 0.1875 0.320426 0.320304 2024-09-16T15:37:41.308 INFO:tasks.workunit.client.0.smithi039.stdout: 12 16 610 594 0.193334 0.207031 0.323818 0.319694 2024-09-16T15:37:42.309 INFO:tasks.workunit.client.0.smithi039.stdout: 13 16 662 646 0.194085 0.203125 0.291698 0.318484 2024-09-16T15:37:43.309 INFO:tasks.workunit.client.0.smithi039.stdout: 14 16 697 681 0.189985 0.136719 0.279126 0.325037 2024-09-16T15:37:44.309 INFO:tasks.workunit.client.0.smithi039.stdout: 15 16 754 738 0.192162 0.222656 0.292734 0.321652 2024-09-16T15:37:45.309 INFO:tasks.workunit.client.0.smithi039.stdout: 16 16 809 793 0.193577 0.214844 0.27541 0.32038 2024-09-16T15:37:46.309 INFO:tasks.workunit.client.0.smithi039.stdout: 17 16 865 849 0.195056 0.21875 0.315183 0.317491 2024-09-16T15:37:47.309 INFO:tasks.workunit.client.0.smithi039.stdout: 18 16 913 897 0.194635 0.1875 0.296518 0.317386 2024-09-16T15:37:48.309 INFO:tasks.workunit.client.0.smithi039.stdout: 19 16 971 955 0.196314 0.226562 0.354152 0.315549 2024-09-16T15:37:49.309 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:37:49.313662+0000 min lat: 0.200282 max lat: 0.779858 avg lat: 0.314493 2024-09-16T15:37:49.310 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:37:49.310 INFO:tasks.workunit.client.0.smithi039.stdout: 20 16 1025 1009 0.197043 0.210938 0.317076 0.314493 2024-09-16T15:37:50.310 INFO:tasks.workunit.client.0.smithi039.stdout: 21 16 1076 1060 0.197145 0.199219 0.236956 0.314856 2024-09-16T15:37:51.310 INFO:tasks.workunit.client.0.smithi039.stdout: 22 16 1124 1108 0.196706 0.1875 0.430734 0.316238 2024-09-16T15:37:52.310 INFO:tasks.workunit.client.0.smithi039.stdout: 23 16 1165 1149 0.195116 0.160156 0.318542 0.318664 2024-09-16T15:37:53.310 INFO:tasks.workunit.client.0.smithi039.stdout: 24 16 1216 1200 0.195285 0.199219 0.286285 0.317809 2024-09-16T15:37:54.310 INFO:tasks.workunit.client.0.smithi039.stdout: 25 16 1261 1245 0.194504 0.175781 0.312091 0.31829 2024-09-16T15:37:55.310 INFO:tasks.workunit.client.0.smithi039.stdout: 26 16 1317 1301 0.195435 0.21875 0.308068 0.318614 2024-09-16T15:37:56.310 INFO:tasks.workunit.client.0.smithi039.stdout: 27 16 1365 1349 0.195141 0.1875 0.297312 0.318284 2024-09-16T15:37:57.311 INFO:tasks.workunit.client.0.smithi039.stdout: 28 16 1421 1405 0.195983 0.21875 0.328307 0.317793 2024-09-16T15:37:58.311 INFO:tasks.workunit.client.0.smithi039.stdout: 29 16 1469 1453 0.19569 0.1875 0.333294 0.317377 2024-09-16T15:37:59.311 INFO:tasks.workunit.client.0.smithi039.stdout: 30 16 1521 1505 0.195936 0.203125 0.307665 0.317282 2024-09-16T15:38:00.311 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 30.2109 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 1521 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4096 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4096 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.196664 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.0239501 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.226562 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.125 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 50 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 6.13123 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 58 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 32 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.316913 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.0575483 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 0.779858 2024-09-16T15:38:00.312 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.200282 2024-09-16T15:38:00.458 INFO:tasks.workunit.client.0.smithi039.stdout:after bench 2024-09-16T15:38:00.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-09-16T15:38:00.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 97482 2024-09-16T15:38:00.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-09-16T15:38:01.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 97482 2024-09-16T15:38:01.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (97482) - No such process 2024-09-16T15:38:01.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-09-16T15:38:01.892 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T15:38:01.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 100028 2024-09-16T15:38:01.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-09-16T15:38:02.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 100028 2024-09-16T15:38:02.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (100028) - No such process 2024-09-16T15:38:02.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-09-16T15:38:03.632 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-16T15:38:03.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 102594 2024-09-16T15:38:03.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-09-16T15:38:04.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 102594 2024-09-16T15:38:04.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (102594) - No such process 2024-09-16T15:38:04.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-09-16T15:38:05.127 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-16T15:38:05.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 105136 2024-09-16T15:38:05.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-09-16T15:38:06.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 105136 2024-09-16T15:38:06.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (105136) - No such process 2024-09-16T15:38:06.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-09-16T15:38:06.767 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 is already down. 2024-09-16T15:38:06.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-16T15:38:08.591 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:38:08.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-16T15:38:10.423 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:38:10.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-16T15:38:12.259 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:38:12.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-16T15:38:14.099 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:38:14.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-16T15:38:14.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-16T15:38:14.166 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:38:15.057 INFO:tasks.workunit.client.0.smithi039.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-16T15:38:15.057 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-16T15:38:15.057 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0x100000000 : using 0x804a8000(2.0 GiB) 2024-09-16T15:38:15.057 INFO:tasks.workunit.client.0.smithi039.stdout:Expanding DB/WAL... 2024-09-16T15:38:15.057 INFO:tasks.workunit.client.0.smithi039.stdout:0 : expanding to 0x536870912 2024-09-16T15:38:15.087 INFO:tasks.workunit.client.0.smithi039.stdout:0 : size label updated to 536870912 2024-09-16T15:38:15.087 INFO:tasks.workunit.client.0.smithi039.stdout:1 : expanding to 0x1073741824 2024-09-16T15:38:15.112 INFO:tasks.workunit.client.0.smithi039.stdout:1 : size label updated to 1073741824 2024-09-16T15:38:15.112 INFO:tasks.workunit.client.0.smithi039.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-09-16T15:38:15.341 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:38:15.342+0000 7fbe87ad2ac0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-09-16T15:38:15.342 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:38:15.342+0000 7fbe87ad2ac0 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-09-16T15:38:20.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-09-16T15:38:20.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-09-16T15:38:20.805 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:38:21.703 INFO:tasks.workunit.client.0.smithi039.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-16T15:38:21.703 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-16T15:38:21.703 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0x101000000 : using 0x814b2000(2.0 GiB) 2024-09-16T15:38:21.703 INFO:tasks.workunit.client.0.smithi039.stdout:Expanding DB/WAL... 2024-09-16T15:38:21.703 INFO:tasks.workunit.client.0.smithi039.stdout:0 : expanding to 0x536870912 2024-09-16T15:38:21.727 INFO:tasks.workunit.client.0.smithi039.stdout:0 : size label updated to 536870912 2024-09-16T15:38:21.727 INFO:tasks.workunit.client.0.smithi039.stdout:1 : expanding to 0x1073741824 2024-09-16T15:38:21.761 INFO:tasks.workunit.client.0.smithi039.stdout:1 : size label updated to 1073741824 2024-09-16T15:38:21.761 INFO:tasks.workunit.client.0.smithi039.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-09-16T15:38:22.037 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:38:22.037+0000 7f7c39ca6ac0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-09-16T15:38:22.037 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:38:22.037+0000 7f7c39ca6ac0 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-09-16T15:38:27.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-09-16T15:38:27.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-09-16T15:38:27.369 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:38:28.291 INFO:tasks.workunit.client.0.smithi039.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-16T15:38:28.292 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-16T15:38:28.292 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0x100020000 : using 0x804e9000(2.0 GiB) 2024-09-16T15:38:28.292 INFO:tasks.workunit.client.0.smithi039.stdout:Expanding DB/WAL... 2024-09-16T15:38:28.292 INFO:tasks.workunit.client.0.smithi039.stdout:0 : expanding to 0x536870912 2024-09-16T15:38:28.319 INFO:tasks.workunit.client.0.smithi039.stdout:0 : size label updated to 536870912 2024-09-16T15:38:28.319 INFO:tasks.workunit.client.0.smithi039.stdout:1 : expanding to 0x1073741824 2024-09-16T15:38:28.352 INFO:tasks.workunit.client.0.smithi039.stdout:1 : size label updated to 1073741824 2024-09-16T15:38:28.352 INFO:tasks.workunit.client.0.smithi039.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-09-16T15:38:28.579 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:38:28.581+0000 7f150a3f8ac0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-09-16T15:38:28.579 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:38:28.581+0000 7f150a3f8ac0 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-09-16T15:38:34.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-09-16T15:38:34.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-09-16T15:38:34.233 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:38:35.128 INFO:tasks.workunit.client.0.smithi039.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-16T15:38:35.128 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-16T15:38:35.128 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0xfff00000 : using 0x803af000(2.0 GiB) 2024-09-16T15:38:35.128 INFO:tasks.workunit.client.0.smithi039.stdout:Expanding DB/WAL... 2024-09-16T15:38:35.128 INFO:tasks.workunit.client.0.smithi039.stdout:0 : expanding to 0x536870912 2024-09-16T15:38:35.168 INFO:tasks.workunit.client.0.smithi039.stdout:0 : size label updated to 536870912 2024-09-16T15:38:35.169 INFO:tasks.workunit.client.0.smithi039.stdout:1 : expanding to 0x1073741824 2024-09-16T15:38:35.193 INFO:tasks.workunit.client.0.smithi039.stdout:1 : size label updated to 1073741824 2024-09-16T15:38:35.194 INFO:tasks.workunit.client.0.smithi039.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-09-16T15:38:35.420 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:38:35.421+0000 7f1cd5248ac0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-09-16T15:38:35.420 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:38:35.421+0000 7f1cd5248ac0 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-09-16T15:38:40.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-16T15:38:42.367 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:38:42.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-16T15:38:44.207 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:38:44.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-16T15:38:46.039 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:38:46.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-16T15:38:47.875 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:38:47.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-16T15:38:47.925 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:38:49.709 INFO:tasks.workunit.client.0.smithi039.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-09-16T15:38:49.709 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-09-16T15:38:49.709 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0x100000000 : using 0x4a8000(4.7 MiB) 2024-09-16T15:38:49.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-09-16T15:38:49.739 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:38:51.519 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-16T15:38:51.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-16T15:38:52.839 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:38:52.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-16T15:38:52.885 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:38:54.668 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-16T15:38:54.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-16T15:38:55.983 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:38:56.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-09-16T15:38:56.029 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:38:57.812 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-16T15:38:57.812 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-16T15:38:57.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-16T15:38:58.631 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:38:58.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-09-16T15:38:58.675 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:38:58.675 INFO:tasks.workunit.client.0.smithi039.stderr:Migrate to WAL device isn't supported. 2024-09-16T15:38:58.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-16T15:38:58.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-16T15:39:00.484 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:39:00.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-16T15:39:00.533 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:39:02.315 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-16T15:39:02.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-16T15:39:03.635 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:39:03.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-09-16T15:39:03.685 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:39:04.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-16T15:39:06.295 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:39:06.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-16T15:39:06.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:39:06.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:39:06.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:39:06.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:39:06.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-16T15:39:06.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:39:06.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:39:06.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:39:06.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:39:06.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-16T15:39:06.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-16T15:39:06.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:39:06.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:39:06.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:39:06.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:39:06.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:39:06.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:39:06.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:39:06.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-16T15:39:06.321 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:39:06.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:39:06.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:39:06.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-16T15:39:06.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:39:06.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:39:06.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:39:06.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:39:06.360 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:06.360+0000 7f32df6a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:06.363 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:06.360+0000 7f32df6a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:06.364 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:06.364+0000 7f32df6a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:06.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:39:06.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:39:06.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:39:06.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:39:06.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:39:06.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:06.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:39:06.797 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:39:06.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:06.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:39:07.153 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:07.152+0000 7f32df6a27c0 -1 Falling back to public interface 2024-09-16T15:39:07.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:08.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:08.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:08.266 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:39:08.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:39:08.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:08.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:39:08.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:09.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:09.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:09.697 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:39:09.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:39:09.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:09.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:39:10.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:11.053 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:11.052+0000 7f32df6a27c0 -1 osd.0 26 log_to_monitors true 2024-09-16T15:39:11.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:11.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:11.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:39:11.112 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:39:11.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:11.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:39:11.551 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 36 up_thru 22 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6802/2019189894,v1:127.0.0.1:6803/2019189894] [v2:127.0.0.1:6804/2019189894,v1:127.0.0.1:6805/2019189894] exists,up 6ae054d5-b804-4b87-94f9-fbc5503745dd 2024-09-16T15:39:11.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:39:11.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:39:11.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:39:11.552 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-16T15:39:11.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=118568 2024-09-16T15:39:11.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-16T15:39:11.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:39:11.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:39:11.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:39:11.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:39:11.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-16T15:39:11.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:39:11.556 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:39:11.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:39:11.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:39:11.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:39:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:39:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:39:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:39:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:39:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:39:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:39:11.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:39:11.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:39:11.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:39:11.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:39:11.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-16T15:39:11.559 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:39:11.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:39:11.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:39:11.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-16T15:39:11.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:39:11.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:39:11.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:39:11.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:39:11.582 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:11.580+0000 7f8632ee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:11.609 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:11.608+0000 7f8632ee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:11.610 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:11.608+0000 7f8632ee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:12.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T15:39:12.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:39:12.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:39:12.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:39:12.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:39:12.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:12.117 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:39:12.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:39:12.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:12.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:39:12.402 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:12.400+0000 7f8632ee87c0 -1 Falling back to public interface 2024-09-16T15:39:12.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:13.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:13.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:13.568 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:39:13.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:39:13.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:13.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:39:13.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:14.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:14.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:14.989 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:39:14.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:39:14.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:14.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:39:15.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:16.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:16.180+0000 7f8632ee87c0 -1 osd.1 28 log_to_monitors true 2024-09-16T15:39:16.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:16.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:16.409 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:39:16.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:39:16.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:16.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:39:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 39 up_thru 27 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/1962680658,v1:127.0.0.1:6811/1962680658] [v2:127.0.0.1:6812/1962680658,v1:127.0.0.1:6813/1962680658] exists,up eb1b948c-057b-433b-8f34-8d78c69ce8c7 2024-09-16T15:39:17.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:39:17.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:39:17.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:39:17.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-16T15:39:17.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=119507 2024-09-16T15:39:17.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-16T15:39:17.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:39:17.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:39:17.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T15:39:17.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:39:17.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-16T15:39:17.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:39:17.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:39:17.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:39:17.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:39:17.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-16T15:39:17.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-16T15:39:17.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:39:17.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:39:17.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:39:17.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:39:17.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:39:17.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:39:17.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:39:17.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:39:17.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-16T15:39:17.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T15:39:17.073 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:39:17.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:39:17.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-16T15:39:17.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T15:39:17.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:39:17.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:39:17.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:39:17.097 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:17.096+0000 7faf581577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:17.099 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:17.100+0000 7faf581577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:17.101 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:17.100+0000 7faf581577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:17.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T15:39:17.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:39:17.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:39:17.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:39:17.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:39:17.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:39:17.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:39:17.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:17.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:39:17.661 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:17.660+0000 7faf581577c0 -1 Falling back to public interface 2024-09-16T15:39:18.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:19.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:19.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:19.103 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:39:19.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:39:19.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:19.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:39:19.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:20.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:20.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:20.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:39:20.532 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:39:20.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:20.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:39:20.690 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:20.688+0000 7faf581577c0 -1 osd.2 31 log_to_monitors true 2024-09-16T15:39:21.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:22.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:22.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:22.029 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:39:22.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:39:22.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:22.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:39:22.602 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 42 up_thru 42 down_at 32 last_clean_interval [15,31) [v2:127.0.0.1:6818/3531836751,v1:127.0.0.1:6819/3531836751] [v2:127.0.0.1:6820/3531836751,v1:127.0.0.1:6821/3531836751] exists,up d3a747ca-52f9-47a9-abe1-7d1cec101ba3 2024-09-16T15:39:22.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:39:22.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:39:22.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:39:22.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-16T15:39:22.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=120446 2024-09-16T15:39:22.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-16T15:39:22.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:39:22.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:39:22.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-16T15:39:22.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:39:22.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-16T15:39:22.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:39:22.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:39:22.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:39:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:39:22.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-16T15:39:22.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-16T15:39:22.610 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T15:39:22.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:39:22.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-16T15:39:22.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-16T15:39:22.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:39:22.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:39:22.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:39:22.634 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:22.632+0000 7fefb486b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:22.636 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:22.636+0000 7fefb486b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:22.637 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:22.636+0000 7fefb486b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:39:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-16T15:39:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:39:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T15:39:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:39:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:39:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:23.217 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:39:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:39:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:39:23.429 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:23.428+0000 7fefb486b7c0 -1 Falling back to public interface 2024-09-16T15:39:23.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:24.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:24.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:24.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:39:24.678 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:39:24.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:24.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:39:25.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:26.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:26.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:26.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:39:26.104 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:39:26.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:26.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:39:26.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:39:27.426 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:39:27.424+0000 7fefb486b7c0 -1 osd.3 33 log_to_monitors true 2024-09-16T15:39:27.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:39:27.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:39:27.519 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:39:27.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:39:27.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:39:27.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:39:28.294 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 46 up_thru 32 down_at 34 last_clean_interval [20,33) [v2:127.0.0.1:6826/909200015,v1:127.0.0.1:6827/909200015] [v2:127.0.0.1:6828/909200015,v1:127.0.0.1:6829/909200015] exists,up 8a48c2fe-87da-456e-960f-2bc0d19675f4 2024-09-16T15:39:28.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:39:28.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:39:28.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:39:28.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-16T15:39:28.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=121380 2024-09-16T15:39:28.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-09-16T15:39:28.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:39:28.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:39:28.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:39:28.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:39:28.298 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:39:28.298 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:39:28.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:39:28.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:39:28.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:39:28.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:39:28.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:39:28.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:39:28.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:39:28.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:39:28.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:39:29.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:39:29.008 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:39:29.008 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T15:39:29.008 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T15:39:29.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:39:29.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:39:29.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:39:29.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=154618822662 2024-09-16T15:39:29.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 154618822662 2024-09-16T15:39:29.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822662' 2024-09-16T15:39:29.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:39:29.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:39:29.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=167503724549 2024-09-16T15:39:29.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 167503724549 2024-09-16T15:39:29.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822662 1-167503724549' 2024-09-16T15:39:29.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:39:29.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:39:29.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=180388626436 2024-09-16T15:39:29.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 180388626436 2024-09-16T15:39:29.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822662 1-167503724549 2-180388626436' 2024-09-16T15:39:29.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:39:29.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T15:39:29.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=197568495619 2024-09-16T15:39:29.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 197568495619 2024-09-16T15:39:29.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822662 1-167503724549 2-180388626436 3-197568495619' 2024-09-16T15:39:29.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:39:29.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-154618822662 2024-09-16T15:39:29.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:39:29.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:39:29.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-154618822662 2024-09-16T15:39:29.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:39:29.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=154618822662 2024-09-16T15:39:29.852 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 154618822662 2024-09-16T15:39:29.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 154618822662' 2024-09-16T15:39:29.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:39:30.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 154618822662 -lt 154618822662 2024-09-16T15:39:30.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:39:30.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-167503724549 2024-09-16T15:39:30.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:39:30.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:39:30.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-167503724549 2024-09-16T15:39:30.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:39:30.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=167503724549 2024-09-16T15:39:30.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 167503724549' 2024-09-16T15:39:30.302 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 167503724549 2024-09-16T15:39:30.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:39:30.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 167503724549 -lt 167503724549 2024-09-16T15:39:30.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:39:30.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-180388626436 2024-09-16T15:39:30.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:39:30.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:39:30.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-180388626436 2024-09-16T15:39:30.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:39:30.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=180388626436 2024-09-16T15:39:30.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 180388626436' 2024-09-16T15:39:30.761 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 180388626436 2024-09-16T15:39:30.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:39:31.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 180388626436 -lt 180388626436 2024-09-16T15:39:31.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:39:31.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-197568495619 2024-09-16T15:39:31.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:39:31.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T15:39:31.189 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-197568495619 2024-09-16T15:39:31.189 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:39:31.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=197568495619 2024-09-16T15:39:31.190 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 197568495619 2024-09-16T15:39:31.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 197568495619' 2024-09-16T15:39:31.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T15:39:31.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 197568495619 -lt 197568495619 2024-09-16T15:39:31.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:39:31.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:39:31.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:39:32.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-16T15:39:32.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:39:32.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:39:32.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:39:32.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:39:32.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:39:32.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:39:32.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:39:32.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-16T15:39:32.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:39:32.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:39:32.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:39:33.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-16T15:39:33.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:39:33.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:39:33.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-16T15:39:33.264 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-16T15:39:33.265 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-16T15:39:33.265 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_122663 2024-09-16T15:39:33.265 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:39:33.265 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-16T15:39:34.265 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 57 41 0.160137 0.160156 0.177107 0.284526 2024-09-16T15:39:35.265 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 110 94 0.183569 0.207031 0.322154 0.301331 2024-09-16T15:39:36.265 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 163 147 0.19138 0.207031 0.311276 0.300943 2024-09-16T15:39:37.265 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 214 198 0.193332 0.199219 0.204427 0.310586 2024-09-16T15:39:38.265 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 263 247 0.19294 0.191406 0.363057 0.31192 2024-09-16T15:39:39.265 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 319 303 0.197237 0.21875 0.166743 0.310611 2024-09-16T15:39:40.266 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 372 356 0.198633 0.207031 0.204296 0.30681 2024-09-16T15:39:41.266 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 422 406 0.198214 0.195312 0.282824 0.30985 2024-09-16T15:39:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 473 457 0.198322 0.199219 0.379136 0.308668 2024-09-16T15:39:43.266 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 525 509 0.1988 0.203125 0.309028 0.308539 2024-09-16T15:39:44.266 INFO:tasks.workunit.client.0.smithi039.stdout: 11 16 581 565 0.20061 0.21875 0.367447 0.307058 2024-09-16T15:39:45.266 INFO:tasks.workunit.client.0.smithi039.stdout: 12 16 635 619 0.201468 0.210938 0.320426 0.307615 2024-09-16T15:39:46.267 INFO:tasks.workunit.client.0.smithi039.stdout: 13 16 680 664 0.199491 0.175781 0.389474 0.305375 2024-09-16T15:39:47.267 INFO:tasks.workunit.client.0.smithi039.stdout: 14 16 731 715 0.199469 0.199219 0.177072 0.310763 2024-09-16T15:39:48.267 INFO:tasks.workunit.client.0.smithi039.stdout: 15 16 775 759 0.197627 0.171875 0.339147 0.311773 2024-09-16T15:39:49.267 INFO:tasks.workunit.client.0.smithi039.stdout: 16 16 829 813 0.198457 0.210938 0.178457 0.311653 2024-09-16T15:39:50.267 INFO:tasks.workunit.client.0.smithi039.stdout: 17 16 881 865 0.19873 0.203125 0.288249 0.311623 2024-09-16T15:39:51.267 INFO:tasks.workunit.client.0.smithi039.stdout: 18 16 936 920 0.199624 0.214844 0.254483 0.310498 2024-09-16T15:39:52.267 INFO:tasks.workunit.client.0.smithi039.stdout: 19 16 981 965 0.198367 0.175781 0.235788 0.313254 2024-09-16T15:39:53.267 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:39:53.271402+0000 min lat: 0.125455 max lat: 0.552666 avg lat: 0.307532 2024-09-16T15:39:53.268 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:39:53.268 INFO:tasks.workunit.client.0.smithi039.stdout: 20 16 1053 1037 0.202509 0.28125 0.218316 0.307532 2024-09-16T15:39:54.268 INFO:tasks.workunit.client.0.smithi039.stdout: 21 16 1125 1109 0.206257 0.28125 0.214339 0.301674 2024-09-16T15:39:55.268 INFO:tasks.workunit.client.0.smithi039.stdout: 22 16 1195 1179 0.209309 0.273438 0.250155 0.296832 2024-09-16T15:39:56.268 INFO:tasks.workunit.client.0.smithi039.stdout: 23 16 1258 1242 0.210907 0.246094 0.279423 0.294462 2024-09-16T15:39:57.268 INFO:tasks.workunit.client.0.smithi039.stdout: 24 16 1314 1298 0.211233 0.21875 0.255035 0.294857 2024-09-16T15:39:58.268 INFO:tasks.workunit.client.0.smithi039.stdout: 25 16 1374 1358 0.212158 0.234375 0.286905 0.292979 2024-09-16T15:39:59.268 INFO:tasks.workunit.client.0.smithi039.stdout: 26 16 1437 1421 0.213462 0.246094 0.203249 0.291264 2024-09-16T15:40:00.269 INFO:tasks.workunit.client.0.smithi039.stdout: 27 16 1500 1484 0.214669 0.246094 0.213297 0.289308 2024-09-16T15:40:01.269 INFO:tasks.workunit.client.0.smithi039.stdout: 28 16 1563 1547 0.21579 0.246094 0.271588 0.28858 2024-09-16T15:40:02.269 INFO:tasks.workunit.client.0.smithi039.stdout: 29 16 1622 1606 0.216295 0.230469 0.290319 0.287394 2024-09-16T15:40:03.269 INFO:tasks.workunit.client.0.smithi039.stdout: 30 16 1686 1670 0.217418 0.25 0.276907 0.286438 2024-09-16T15:40:04.269 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 30.2017 2024-09-16T15:40:04.269 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 1686 2024-09-16T15:40:04.269 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4096 2024-09-16T15:40:04.269 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4096 2024-09-16T15:40:04.269 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.218065 2024-09-16T15:40:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.031019 2024-09-16T15:40:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.28125 2024-09-16T15:40:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.160156 2024-09-16T15:40:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 55 2024-09-16T15:40:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 7.94087 2024-09-16T15:40:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 72 2024-09-16T15:40:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 41 2024-09-16T15:40:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.285968 2024-09-16T15:40:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.0664026 2024-09-16T15:40:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 0.552666 2024-09-16T15:40:04.270 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.125455 2024-09-16T15:40:04.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 118568 2024-09-16T15:40:04.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-09-16T15:40:05.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 118568 2024-09-16T15:40:05.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (118568) - No such process 2024-09-16T15:40:05.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-09-16T15:40:05.798 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T15:40:05.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 119507 2024-09-16T15:40:05.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-09-16T15:40:06.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 119507 2024-09-16T15:40:06.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (119507) - No such process 2024-09-16T15:40:06.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-09-16T15:40:07.350 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-16T15:40:07.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 120446 2024-09-16T15:40:07.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-09-16T15:40:08.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 120446 2024-09-16T15:40:08.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (120446) - No such process 2024-09-16T15:40:08.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-09-16T15:40:08.829 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-16T15:40:08.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 121380 2024-09-16T15:40:08.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-09-16T15:40:09.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 121380 2024-09-16T15:40:09.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (121380) - No such process 2024-09-16T15:40:09.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-09-16T15:40:10.522 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 is already down. 2024-09-16T15:40:10.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-16T15:40:11.844 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:40:11.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-09-16T15:40:12.168 INFO:tasks.workunit.client.0.smithi039.stderr:512+0 records in 2024-09-16T15:40:12.169 INFO:tasks.workunit.client.0.smithi039.stderr:512+0 records out 2024-09-16T15:40:12.169 INFO:tasks.workunit.client.0.smithi039.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.301005 s, 1.8 GB/s 2024-09-16T15:40:12.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-09-16T15:40:12.193 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:40:16.732 INFO:tasks.workunit.client.0.smithi039.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-09-16T15:40:16.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-16T15:40:18.548 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:40:18.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-16T15:40:19.876 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:40:19.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-09-16T15:40:20.481 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-16T15:40:20.481 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-16T15:40:20.481 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.586711 s, 1.8 GB/s 2024-09-16T15:40:20.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-09-16T15:40:20.505 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:40:28.540 INFO:tasks.workunit.client.0.smithi039.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-09-16T15:40:28.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-09-16T15:40:28.571 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:40:30.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-16T15:40:32.180 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:40:32.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-16T15:40:33.004 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:40:33.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-09-16T15:40:33.040 INFO:tasks.workunit.client.0.smithi039.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-16T15:40:33.040 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:40:33.832 INFO:tasks.workunit.client.0.smithi039.stdout:DB device added 2024-09-16T15:40:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-09-16T15:40:33.864 INFO:tasks.workunit.client.0.smithi039.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-16T15:40:33.864 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:40:35.152 INFO:tasks.workunit.client.0.smithi039.stdout:WAL device added 2024-09-16T15:40:35.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-09-16T15:40:35.185 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:40:37.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-16T15:40:39.288 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:40:39.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-16T15:40:40.608 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:40:40.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-09-16T15:40:40.649 INFO:tasks.workunit.client.0.smithi039.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-16T15:40:40.649 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:40:41.940 INFO:tasks.workunit.client.0.smithi039.stdout:WAL device added 2024-09-16T15:40:41.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-16T15:40:41.972 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:40:44.020 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-09-16T15:40:44.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-16T15:40:45.348 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:40:45.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-16T15:40:45.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:40:45.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:40:45.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:40:45.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:40:45.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-16T15:40:45.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:40:45.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:40:45.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:40:45.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:40:45.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-16T15:40:45.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-16T15:40:45.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:40:45.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:40:45.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:40:45.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:40:45.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:40:45.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:40:45.369 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:40:45.369 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:40:45.369 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:40:45.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:40:45.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:40:45.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:40:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:40:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:40:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:40:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:40:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:40:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:40:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:40:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:40:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-16T15:40:45.372 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:40:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:40:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:40:45.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-16T15:40:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:40:45.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:40:45.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:40:45.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:40:45.407 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:45.406+0000 7fa9c60437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:40:45.410 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:45.410+0000 7fa9c60437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:40:45.412 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:45.414+0000 7fa9c60437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:40:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:40:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:40:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:40:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:40:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:40:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:40:45.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:40:45.854 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:40:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:40:45.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:40:46.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:40:46.451 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:46.450+0000 7fa9c60437c0 -1 Falling back to public interface 2024-09-16T15:40:47.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:40:47.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:40:47.321 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:40:47.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:40:47.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:40:47.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:40:47.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:40:48.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:40:48.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:40:48.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:40:48.753 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:40:48.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:40:48.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:40:49.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:40:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:40:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:40:50.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:40:50.166 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:40:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:40:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:40:50.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:40:51.334 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:51.334+0000 7fa9c60437c0 -1 osd.0 48 log_to_monitors true 2024-09-16T15:40:51.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:40:51.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:40:51.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:40:51.658 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:40:51.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:40:51.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:40:52.253 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 58 up_thru 58 down_at 49 last_clean_interval [36,48) [v2:127.0.0.1:6802/2127929205,v1:127.0.0.1:6803/2127929205] [v2:127.0.0.1:6804/2127929205,v1:127.0.0.1:6805/2127929205] exists,up 6ae054d5-b804-4b87-94f9-fbc5503745dd 2024-09-16T15:40:52.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:40:52.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:40:52.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:40:52.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-16T15:40:52.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=128772 2024-09-16T15:40:52.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-16T15:40:52.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:40:52.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:40:52.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:40:52.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:40:52.260 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:40:52.260 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:40:52.260 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:40:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-16T15:40:52.262 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:40:52.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:40:52.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:40:52.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-16T15:40:52.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:40:52.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:40:52.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:40:52.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:40:52.292 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:52.294+0000 7f5a2266b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:40:52.325 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:52.326+0000 7f5a2266b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:40:52.326 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:52.326+0000 7f5a2266b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:40:52.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T15:40:52.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:40:52.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:40:52.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:40:52.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:40:52.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:40:52.700 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:40:52.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:40:52.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:40:52.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:40:53.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:40:53.363 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:53.362+0000 7f5a2266b7c0 -1 Falling back to public interface 2024-09-16T15:40:54.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:40:54.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:40:54.167 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:40:54.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:40:54.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:40:54.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:40:54.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:40:55.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:40:55.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:40:55.640 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:40:55.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:40:55.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:40:55.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:40:56.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:40:57.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:40:57.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:40:57.088 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:40:57.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:40:57.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:40:57.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:40:57.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:40:58.163 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:58.162+0000 7f5a2266b7c0 -1 osd.1 50 log_to_monitors true 2024-09-16T15:40:58.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:40:58.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:40:58.503 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:40:58.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:40:58.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:40:58.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:40:59.008 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 61 up_thru 61 down_at 51 last_clean_interval [39,50) [v2:127.0.0.1:6810/2516384162,v1:127.0.0.1:6811/2516384162] [v2:127.0.0.1:6812/2516384162,v1:127.0.0.1:6813/2516384162] exists,up eb1b948c-057b-433b-8f34-8d78c69ce8c7 2024-09-16T15:40:59.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:40:59.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:40:59.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:40:59.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=129741 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:40:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-16T15:40:59.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-16T15:40:59.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:40:59.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:40:59.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:40:59.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:40:59.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:40:59.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:40:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-16T15:40:59.016 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:40:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T15:40:59.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:40:59.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-16T15:40:59.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T15:40:59.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:40:59.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:40:59.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:40:59.046 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:59.046+0000 7f7d73de47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:40:59.221 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:59.222+0000 7f7d73de47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:40:59.243 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:40:59.242+0000 7f7d73de47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:40:59.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T15:40:59.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:40:59.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:40:59.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:40:59.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:40:59.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:40:59.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:40:59.532 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:40:59.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:40:59.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:40:59.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:41:00.283 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:41:00.281+0000 7f7d73de47c0 -1 Falling back to public interface 2024-09-16T15:41:00.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:41:00.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:41:00.982 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:41:00.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:41:00.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:41:00.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:41:01.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:41:02.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:41:02.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:41:02.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:41:02.405 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:41:02.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:41:02.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:41:02.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:41:03.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:41:03.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:41:03.839 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:41:03.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:41:03.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:41:03.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:41:04.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:41:05.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:41:05.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:41:05.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:41:05.275 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:41:05.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:41:05.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:41:05.362 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:41:05.361+0000 7f7d73de47c0 -1 osd.2 53 log_to_monitors true 2024-09-16T15:41:05.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:41:06.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:41:06.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:41:06.781 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:41:06.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:41:06.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:41:06.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:41:07.264 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 65 up_thru 65 down_at 54 last_clean_interval [42,53) [v2:127.0.0.1:6818/2727616426,v1:127.0.0.1:6819/2727616426] [v2:127.0.0.1:6820/2727616426,v1:127.0.0.1:6821/2727616426] exists,up d3a747ca-52f9-47a9-abe1-7d1cec101ba3 2024-09-16T15:41:07.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:41:07.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:41:07.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:41:07.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-16T15:41:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=130730 2024-09-16T15:41:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:41:07.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:41:07.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:41:07.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:41:07.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:41:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:41:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:41:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-16T15:41:07.271 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T15:41:07.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-16T15:41:07.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:41:07.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-16T15:41:07.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-16T15:41:07.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:41:07.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:41:07.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:41:07.306 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:41:07.305+0000 7f09396c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:41:07.308 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:41:07.309+0000 7f09396c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:41:07.309 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:41:07.309+0000 7f09396c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:41:07.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-16T15:41:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:41:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T15:41:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:41:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:41:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:41:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:41:07.687 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:41:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:41:07.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:41:08.102 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:41:08.101+0000 7f09396c37c0 -1 Falling back to public interface 2024-09-16T15:41:08.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:41:09.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:41:09.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:41:09.122 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:41:09.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:41:09.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:41:09.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:41:09.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:41:10.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:41:10.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:41:10.542 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:41:10.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:41:10.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:41:10.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:41:10.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:41:11.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:41:11.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:41:11.967 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:41:11.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:41:11.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:41:11.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:41:12.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:41:12.774 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:41:12.773+0000 7f09396c37c0 -1 osd.3 55 log_to_monitors true 2024-09-16T15:41:13.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:41:13.501 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:41:13.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:41:13.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:41:13.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:41:13.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:41:14.168 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 68 up_thru 54 down_at 56 last_clean_interval [46,55) [v2:127.0.0.1:6826/78817654,v1:127.0.0.1:6827/78817654] [v2:127.0.0.1:6828/78817654,v1:127.0.0.1:6829/78817654] exists,up 8a48c2fe-87da-456e-960f-2bc0d19675f4 2024-09-16T15:41:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:41:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:41:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:41:14.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-16T15:41:14.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=131705 2024-09-16T15:41:14.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-16T15:41:14.203 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-16T15:41:14.203 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-16T15:41:14.204 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_132654 2024-09-16T15:41:14.204 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:41:14.204 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-16T15:41:15.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 23 7 0.0273408 0.0273438 0.738944 0.423905 2024-09-16T15:41:16.204 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 72 56 0.10936 0.191406 0.336943 0.533597 2024-09-16T15:41:17.204 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 121 105 0.136699 0.191406 0.345829 0.435394 2024-09-16T15:41:18.204 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 172 156 0.152321 0.199219 0.363158 0.39376 2024-09-16T15:41:19.204 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 215 199 0.155446 0.167969 0.346434 0.382592 2024-09-16T15:41:20.204 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 260 244 0.15883 0.175781 0.384035 0.381161 2024-09-16T15:41:21.204 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 309 293 0.16348 0.191406 0.315918 0.373163 2024-09-16T15:41:22.205 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 361 345 0.168432 0.203125 0.235269 0.365166 2024-09-16T15:41:23.205 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 407 391 0.169681 0.179688 0.31052 0.361666 2024-09-16T15:41:24.205 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 456 440 0.171851 0.191406 0.327308 0.35696 2024-09-16T15:41:25.205 INFO:tasks.workunit.client.0.smithi039.stdout: 11 16 497 481 0.170786 0.160156 0.332991 0.361429 2024-09-16T15:41:26.205 INFO:tasks.workunit.client.0.smithi039.stdout: 12 16 542 526 0.1712 0.175781 0.35668 0.359167 2024-09-16T15:41:27.205 INFO:tasks.workunit.client.0.smithi039.stdout: 13 16 594 578 0.173654 0.203125 0.374948 0.357618 2024-09-16T15:41:28.205 INFO:tasks.workunit.client.0.smithi039.stdout: 14 16 634 618 0.172409 0.15625 0.21695 0.355885 2024-09-16T15:41:29.206 INFO:tasks.workunit.client.0.smithi039.stdout: 15 16 685 669 0.174195 0.199219 0.350802 0.354171 2024-09-16T15:41:30.206 INFO:tasks.workunit.client.0.smithi039.stdout: 16 16 724 708 0.172828 0.152344 0.440009 0.357235 2024-09-16T15:41:31.206 INFO:tasks.workunit.client.0.smithi039.stdout: 17 16 771 755 0.173459 0.183594 0.341098 0.35703 2024-09-16T15:41:32.206 INFO:tasks.workunit.client.0.smithi039.stdout: 18 16 814 798 0.173153 0.167969 0.394135 0.35736 2024-09-16T15:41:33.206 INFO:tasks.workunit.client.0.smithi039.stdout: 19 16 864 848 0.174318 0.195312 0.317291 0.356216 2024-09-16T15:41:34.206 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:41:34.209489+0000 min lat: 0.192261 max lat: 1.23456 avg lat: 0.353371 2024-09-16T15:41:34.206 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:41:34.206 INFO:tasks.workunit.client.0.smithi039.stdout: 20 16 915 899 0.175562 0.199219 0.210292 0.353371 2024-09-16T15:41:35.206 INFO:tasks.workunit.client.0.smithi039.stdout: 21 16 966 950 0.176686 0.199219 0.225842 0.350901 2024-09-16T15:41:36.207 INFO:tasks.workunit.client.0.smithi039.stdout: 22 16 1008 992 0.176111 0.164062 0.224559 0.350337 2024-09-16T15:41:37.207 INFO:tasks.workunit.client.0.smithi039.stdout: 23 16 1060 1044 0.177284 0.203125 0.22241 0.350334 2024-09-16T15:41:38.207 INFO:tasks.workunit.client.0.smithi039.stdout: 24 16 1098 1082 0.176082 0.148438 0.316482 0.35154 2024-09-16T15:41:39.207 INFO:tasks.workunit.client.0.smithi039.stdout: 25 16 1146 1130 0.176537 0.1875 0.375526 0.352099 2024-09-16T15:41:40.207 INFO:tasks.workunit.client.0.smithi039.stdout: 26 16 1178 1162 0.174555 0.125 0.627318 0.353772 2024-09-16T15:41:41.207 INFO:tasks.workunit.client.0.smithi039.stdout: 27 16 1222 1206 0.174454 0.171875 0.237103 0.356655 2024-09-16T15:41:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: 28 16 1262 1246 0.173803 0.15625 0.418877 0.356941 2024-09-16T15:41:43.208 INFO:tasks.workunit.client.0.smithi039.stdout: 29 16 1306 1290 0.173736 0.171875 0.366548 0.357554 2024-09-16T15:41:44.208 INFO:tasks.workunit.client.0.smithi039.stdout: 30 16 1353 1337 0.174063 0.183594 0.374613 0.356748 2024-09-16T15:41:45.208 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 30.3208 2024-09-16T15:41:45.208 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 1353 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4096 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4096 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.174308 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.033832 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.203125 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.0273438 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 44 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 8.66098 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 52 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 7 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.357024 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.10556 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 1.23456 2024-09-16T15:41:45.209 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.192261 2024-09-16T15:41:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 128772 2024-09-16T15:41:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-09-16T15:41:46.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 128772 2024-09-16T15:41:46.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (128772) - No such process 2024-09-16T15:41:46.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-09-16T15:41:47.008 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T15:41:47.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 129741 2024-09-16T15:41:47.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-09-16T15:41:48.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 129741 2024-09-16T15:41:48.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (129741) - No such process 2024-09-16T15:41:48.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-09-16T15:41:48.495 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-16T15:41:48.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 130730 2024-09-16T15:41:48.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-09-16T15:41:49.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 130730 2024-09-16T15:41:49.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (130730) - No such process 2024-09-16T15:41:49.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-09-16T15:41:49.933 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-16T15:41:49.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 131705 2024-09-16T15:41:49.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-09-16T15:41:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 131705 2024-09-16T15:41:50.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (131705) - No such process 2024-09-16T15:41:50.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-09-16T15:41:51.512 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 is already down. 2024-09-16T15:41:51.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-16T15:41:53.336 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:41:53.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-09-16T15:41:53.909 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-16T15:41:53.909 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-16T15:41:53.909 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.547059 s, 2.0 GB/s 2024-09-16T15:41:53.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-09-16T15:41:53.934 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:42:02.968 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-16T15:42:02.969 INFO:tasks.workunit.client.0.smithi039.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-09-16T15:42:02.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-16T15:42:04.792 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:42:04.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-09-16T15:42:05.115 INFO:tasks.workunit.client.0.smithi039.stderr:512+0 records in 2024-09-16T15:42:05.115 INFO:tasks.workunit.client.0.smithi039.stderr:512+0 records out 2024-09-16T15:42:05.115 INFO:tasks.workunit.client.0.smithi039.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.301324 s, 1.8 GB/s 2024-09-16T15:42:05.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-09-16T15:42:05.139 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:42:10.432 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-16T15:42:10.433 INFO:tasks.workunit.client.0.smithi039.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-09-16T15:42:10.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-09-16T15:42:10.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-16T15:42:12.336 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:42:12.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-16T15:42:14.168 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:42:14.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-09-16T15:42:14.776 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-16T15:42:14.776 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-16T15:42:14.776 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.583102 s, 1.8 GB/s 2024-09-16T15:42:14.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-09-16T15:42:14.802 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:42:23.845 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-16T15:42:23.845 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-09-16T15:42:23.845 INFO:tasks.workunit.client.0.smithi039.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-09-16T15:42:23.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-09-16T15:42:23.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-16T15:42:25.276 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:42:25.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-16T15:42:25.321 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:42:26.856 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-16T15:42:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-09-16T15:42:26.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-16T15:42:27.796 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:42:27.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-09-16T15:42:29.624 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:42:29.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-16T15:42:30.226 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-16T15:42:30.226 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-16T15:42:30.226 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.575642 s, 1.9 GB/s 2024-09-16T15:42:30.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-16T15:42:30.250 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:42:30.250 INFO:tasks.workunit.client.0.smithi039.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-09-16T15:42:30.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-16T15:42:30.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-16T15:42:32.056 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:42:32.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-16T15:42:39.375 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-16T15:42:39.375 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-16T15:42:39.375 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.16188 s, 150 MB/s 2024-09-16T15:42:39.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-16T15:42:39.401 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:42:41.948 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-16T15:42:41.949 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-16T15:42:41.949 INFO:tasks.workunit.client.0.smithi039.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-09-16T15:42:41.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-16T15:42:43.264 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:42:43.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-09-16T15:42:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-16T15:42:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-16T15:42:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.569053 s, 1.9 GB/s 2024-09-16T15:42:43.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-09-16T15:42:43.874 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:42:52.160 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-09-16T15:42:52.161 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-16T15:42:52.161 INFO:tasks.workunit.client.0.smithi039.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-09-16T15:42:52.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-16T15:42:53.481 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:42:53.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-16T15:42:53.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:42:53.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:42:53.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:42:53.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:42:53.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-16T15:42:53.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:42:53.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:42:53.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:42:53.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:42:53.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-16T15:42:53.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-16T15:42:53.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:42:53.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:42:53.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:42:53.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:42:53.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:42:53.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:42:53.504 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:42:53.504 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:42:53.504 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:42:53.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:42:53.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:42:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:42:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:42:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:42:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:42:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:42:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:42:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:42:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:42:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:42:53.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-16T15:42:53.506 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:42:53.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:42:53.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:42:53.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-16T15:42:53.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:42:53.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:42:53.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:42:53.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:42:53.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:42:53.543+0000 7f99cd1387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:42:53.544 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:42:53.547+0000 7f99cd1387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:42:53.546 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:42:53.547+0000 7f99cd1387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:42:53.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:42:53.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:42:53.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:42:53.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:42:53.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:42:53.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:42:53.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:42:53.930 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:42:53.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:42:53.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:42:54.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:42:54.579 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:42:54.579+0000 7f99cd1387c0 -1 Falling back to public interface 2024-09-16T15:42:55.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:42:55.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:42:55.373 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:42:55.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:42:55.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:42:55.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:42:55.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:42:56.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:42:56.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:42:56.824 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:42:56.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:42:56.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:42:56.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:42:57.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:42:58.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:42:58.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:42:58.249 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:42:58.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:42:58.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:42:58.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:42:58.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:42:59.356 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:42:59.355+0000 7f99cd1387c0 -1 osd.0 70 log_to_monitors true 2024-09-16T15:42:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:42:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:42:59.677 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:42:59.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:42:59.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:42:59.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:43:00.167 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 80 up_thru 80 down_at 71 last_clean_interval [58,70) [v2:127.0.0.1:6802/579351051,v1:127.0.0.1:6803/579351051] [v2:127.0.0.1:6804/579351051,v1:127.0.0.1:6805/579351051] exists,up 6ae054d5-b804-4b87-94f9-fbc5503745dd 2024-09-16T15:43:00.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:43:00.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:43:00.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:43:00.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-16T15:43:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=138784 2024-09-16T15:43:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-16T15:43:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:43:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:43:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:43:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:43:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-16T15:43:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:43:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:43:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:43:00.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:43:00.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-16T15:43:00.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-16T15:43:00.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:43:00.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:43:00.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:43:00.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:43:00.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:43:00.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:43:00.173 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:43:00.173 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:43:00.173 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:43:00.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:43:00.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:43:00.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:43:00.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:43:00.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:43:00.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:43:00.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:43:00.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:43:00.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:43:00.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:43:00.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:43:00.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-16T15:43:00.174 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:43:00.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:43:00.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:43:00.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-16T15:43:00.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:43:00.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:43:00.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:43:00.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:43:00.205 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:00.207+0000 7fd31634e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:00.207 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:00.207+0000 7fd31634e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:00.208 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:00.211+0000 7fd31634e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:00.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T15:43:00.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:43:00.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:43:00.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:43:00.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:43:00.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:00.644 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:43:00.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:43:00.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:00.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:43:00.774 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:00.775+0000 7fd31634e7c0 -1 Falling back to public interface 2024-09-16T15:43:01.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:43:02.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:43:02.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:02.085 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:43:02.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:43:02.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:02.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:43:02.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:43:03.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:43:03.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:03.577 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:43:03.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:43:03.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:03.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:43:03.967 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:03.966+0000 7fd31634e7c0 -1 osd.1 73 log_to_monitors true 2024-09-16T15:43:03.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:43:04.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:43:04.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:04.995 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:43:04.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:43:04.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:04.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:43:05.422 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 83 up_thru 83 down_at 74 last_clean_interval [61,73) [v2:127.0.0.1:6810/410481726,v1:127.0.0.1:6811/410481726] [v2:127.0.0.1:6812/410481726,v1:127.0.0.1:6813/410481726] exists,up eb1b948c-057b-433b-8f34-8d78c69ce8c7 2024-09-16T15:43:05.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:43:05.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:43:05.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:43:05.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=139753 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:43:05.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-16T15:43:05.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-16T15:43:05.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:43:05.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:43:05.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:43:05.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:43:05.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:43:05.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:43:05.427 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:43:05.427 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:43:05.427 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:43:05.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:43:05.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:43:05.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:43:05.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:43:05.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:43:05.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:43:05.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:43:05.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:43:05.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:43:05.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:43:05.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:43:05.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-16T15:43:05.429 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:43:05.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T15:43:05.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:43:05.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-16T15:43:05.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T15:43:05.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:43:05.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:43:05.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:43:05.459 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:05.458+0000 7fec6ffe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:05.474 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:05.474+0000 7fec6ffe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:05.475 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:05.474+0000 7fec6ffe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:05.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T15:43:05.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:43:05.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:43:05.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:43:05.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:43:05.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:05.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:43:05.851 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:43:05.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:05.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:43:06.271 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:06.270+0000 7fec6ffe47c0 -1 Falling back to public interface 2024-09-16T15:43:06.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:43:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:43:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:07.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:43:07.287 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:43:07.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:07.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:43:07.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:43:08.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:43:08.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:08.702 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:43:08.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:43:08.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:08.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:43:09.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:43:10.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:43:10.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:10.166 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:43:10.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:43:10.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:10.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:43:10.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:10.542+0000 7fec6ffe47c0 -1 osd.2 75 log_to_monitors true 2024-09-16T15:43:10.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:43:11.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:43:11.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:11.673 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:43:11.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:43:11.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:11.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:43:12.442 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 86 up_thru 86 down_at 76 last_clean_interval [65,75) [v2:127.0.0.1:6818/1936434360,v1:127.0.0.1:6819/1936434360] [v2:127.0.0.1:6820/1936434360,v1:127.0.0.1:6821/1936434360] exists,up d3a747ca-52f9-47a9-abe1-7d1cec101ba3 2024-09-16T15:43:12.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:43:12.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:43:12.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:43:12.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-16T15:43:12.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=140687 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:43:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:43:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:43:12.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:43:12.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:43:12.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:43:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:43:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:43:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:43:12.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:43:12.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:43:12.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:43:12.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:43:12.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:43:12.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:43:12.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:43:12.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:43:12.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:43:12.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:43:12.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:43:12.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-16T15:43:12.450 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T15:43:12.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-16T15:43:12.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:43:12.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-16T15:43:12.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-16T15:43:12.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:43:12.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:43:12.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:43:12.474 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:12.474+0000 7f7eb07c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:12.490 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:12.490+0000 7f7eb07c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:12.492 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:12.490+0000 7f7eb07c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:12.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-16T15:43:12.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:43:12.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T15:43:12.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:43:12.913 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:43:12.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:43:12.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:12.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:43:12.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:12.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:43:13.303 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:13.302+0000 7f7eb07c27c0 -1 Falling back to public interface 2024-09-16T15:43:13.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:43:14.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:43:14.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:14.360 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:43:14.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:43:14.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:14.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:43:14.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:43:15.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:43:15.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:15.777 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:43:15.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:43:15.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:15.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:43:16.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:43:17.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:43:17.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:17.205 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:43:17.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:43:17.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:17.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:43:17.258 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:17.258+0000 7f7eb07c27c0 -1 osd.3 77 log_to_monitors true 2024-09-16T15:43:17.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:43:18.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:43:18.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:43:18.768 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:43:18.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:43:18.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:43:18.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T15:43:19.262 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 90 up_thru 90 down_at 78 last_clean_interval [68,77) [v2:127.0.0.1:6826/2852830333,v1:127.0.0.1:6827/2852830333] [v2:127.0.0.1:6828/2852830333,v1:127.0.0.1:6829/2852830333] exists,up 8a48c2fe-87da-456e-960f-2bc0d19675f4 2024-09-16T15:43:19.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:43:19.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:43:19.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:43:19.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-16T15:43:19.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=141651 2024-09-16T15:43:19.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-16T15:43:19.310 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-16T15:43:19.311 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-16T15:43:19.311 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_142600 2024-09-16T15:43:19.311 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:43:19.311 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-16T15:43:20.311 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 113 97 0.378879 0.378906 0.196897 0.160395 2024-09-16T15:43:21.311 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 227 211 0.412072 0.445312 0.133313 0.149115 2024-09-16T15:43:22.311 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 342 326 0.424438 0.449219 0.166094 0.144431 2024-09-16T15:43:23.311 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 445 429 0.418901 0.402344 0.115563 0.145518 2024-09-16T15:43:24.311 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 554 538 0.420265 0.425781 0.176057 0.145739 2024-09-16T15:43:25.311 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 643 627 0.408154 0.347656 0.174285 0.150631 2024-09-16T15:43:26.312 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 739 723 0.403409 0.375 0.162333 0.151886 2024-09-16T15:43:27.312 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 863 847 0.413523 0.484375 0.0888389 0.149938 2024-09-16T15:43:28.312 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 995 979 0.424859 0.515625 0.144424 0.14572 2024-09-16T15:43:29.312 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 1081 1065 0.415964 0.335938 0.10489 0.149052 2024-09-16T15:43:30.312 INFO:tasks.workunit.client.0.smithi039.stdout: 11 16 1204 1188 0.421822 0.480469 0.161171 0.147144 2024-09-16T15:43:31.312 INFO:tasks.workunit.client.0.smithi039.stdout: 12 16 1327 1311 0.426705 0.480469 0.11841 0.145888 2024-09-16T15:43:32.312 INFO:tasks.workunit.client.0.smithi039.stdout: 13 16 1447 1431 0.429934 0.46875 0.161699 0.144668 2024-09-16T15:43:33.313 INFO:tasks.workunit.client.0.smithi039.stdout: 14 16 1545 1529 0.426566 0.382812 0.115723 0.145344 2024-09-16T15:43:34.313 INFO:tasks.workunit.client.0.smithi039.stdout: 15 16 1648 1632 0.424949 0.402344 0.256209 0.144664 2024-09-16T15:43:35.313 INFO:tasks.workunit.client.0.smithi039.stdout: 16 16 1733 1717 0.41914 0.332031 0.16056 0.148409 2024-09-16T15:43:36.313 INFO:tasks.workunit.client.0.smithi039.stdout: 17 16 1845 1829 0.420217 0.4375 0.170876 0.147922 2024-09-16T15:43:37.313 INFO:tasks.workunit.client.0.smithi039.stdout: 18 16 1957 1941 0.421175 0.4375 0.120707 0.147922 2024-09-16T15:43:38.313 INFO:tasks.workunit.client.0.smithi039.stdout: 19 16 2063 2047 0.420798 0.414062 0.137356 0.147756 2024-09-16T15:43:39.313 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:43:39.315882+0000 min lat: 0.0715452 max lat: 0.531579 avg lat: 0.14797 2024-09-16T15:43:39.313 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:43:39.313 INFO:tasks.workunit.client.0.smithi039.stdout: 20 16 2174 2158 0.421436 0.433594 0.166026 0.14797 2024-09-16T15:43:40.313 INFO:tasks.workunit.client.0.smithi039.stdout: 21 16 2270 2254 0.419222 0.375 0.113777 0.148916 2024-09-16T15:43:41.314 INFO:tasks.workunit.client.0.smithi039.stdout: 22 16 2370 2354 0.41792 0.390625 0.134665 0.148878 2024-09-16T15:43:42.313 INFO:tasks.workunit.client.0.smithi039.stdout: 23 16 2489 2473 0.419957 0.464844 0.111338 0.148165 2024-09-16T15:43:43.314 INFO:tasks.workunit.client.0.smithi039.stdout: 24 16 2609 2593 0.421987 0.46875 0.160408 0.147631 2024-09-16T15:43:44.314 INFO:tasks.workunit.client.0.smithi039.stdout: 25 16 2712 2696 0.4212 0.402344 0.151097 0.14776 2024-09-16T15:43:45.314 INFO:tasks.workunit.client.0.smithi039.stdout: 26 16 2801 2785 0.418369 0.347656 0.202569 0.14791 2024-09-16T15:43:46.314 INFO:tasks.workunit.client.0.smithi039.stdout: 27 16 2904 2888 0.417774 0.402344 0.124826 0.149199 2024-09-16T15:43:47.314 INFO:tasks.workunit.client.0.smithi039.stdout: 28 16 3022 3006 0.419313 0.460938 0.0891306 0.148608 2024-09-16T15:43:48.314 INFO:tasks.workunit.client.0.smithi039.stdout: 29 16 3134 3118 0.419938 0.4375 0.181293 0.148462 2024-09-16T15:43:49.314 INFO:tasks.workunit.client.0.smithi039.stdout: 30 16 3254 3238 0.421563 0.46875 0.116667 0.148034 2024-09-16T15:43:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 30.0972 2024-09-16T15:43:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 3254 2024-09-16T15:43:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4096 2024-09-16T15:43:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4096 2024-09-16T15:43:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.422329 2024-09-16T15:43:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.0486046 2024-09-16T15:43:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.515625 2024-09-16T15:43:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.332031 2024-09-16T15:43:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 108 2024-09-16T15:43:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 12.4428 2024-09-16T15:43:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 132 2024-09-16T15:43:50.316 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 85 2024-09-16T15:43:50.316 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.147857 2024-09-16T15:43:50.316 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.0484934 2024-09-16T15:43:50.316 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 0.531579 2024-09-16T15:43:50.316 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.0676551 2024-09-16T15:43:50.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-09-16T15:43:50.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:43:50.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:43:50.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:43:50.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:43:50.404 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:43:50.404 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:43:50.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:43:50.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:43:50.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:43:50.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:43:50.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:43:50.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:43:50.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:43:50.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:43:50.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:43:50.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:43:50.955 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:43:50.955 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T15:43:50.955 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T15:43:50.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:43:50.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:43:50.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:43:51.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=343597383693 2024-09-16T15:43:51.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 343597383693 2024-09-16T15:43:51.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-343597383693' 2024-09-16T15:43:51.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:43:51.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:43:51.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=356482285580 2024-09-16T15:43:51.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 356482285580 2024-09-16T15:43:51.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-343597383693 1-356482285580' 2024-09-16T15:43:51.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:43:51.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:43:51.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=369367187467 2024-09-16T15:43:51.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 369367187467 2024-09-16T15:43:51.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-343597383693 1-356482285580 2-369367187467' 2024-09-16T15:43:51.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:43:51.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T15:43:51.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=386547056649 2024-09-16T15:43:51.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 386547056649 2024-09-16T15:43:51.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-343597383693 1-356482285580 2-369367187467 3-386547056649' 2024-09-16T15:43:51.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:43:51.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-343597383693 2024-09-16T15:43:51.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:43:51.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:43:51.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-343597383693 2024-09-16T15:43:51.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:43:51.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=343597383693 2024-09-16T15:43:51.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 343597383693' 2024-09-16T15:43:51.567 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 343597383693 2024-09-16T15:43:51.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:43:52.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 343597383691 -lt 343597383693 2024-09-16T15:43:52.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:43:53.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:43:53.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:43:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 343597383693 -lt 343597383693 2024-09-16T15:43:53.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:43:53.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-356482285580 2024-09-16T15:43:53.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:43:53.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:43:53.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-356482285580 2024-09-16T15:43:53.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:43:53.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=356482285580 2024-09-16T15:43:53.462 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 356482285580 2024-09-16T15:43:53.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 356482285580' 2024-09-16T15:43:53.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:43:53.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 356482285580 -lt 356482285580 2024-09-16T15:43:53.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:43:53.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-369367187467 2024-09-16T15:43:53.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:43:53.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:43:53.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-369367187467 2024-09-16T15:43:53.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:43:53.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=369367187467 2024-09-16T15:43:53.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 369367187467' 2024-09-16T15:43:53.888 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 369367187467 2024-09-16T15:43:53.889 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:43:54.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 369367187467 -lt 369367187467 2024-09-16T15:43:54.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:43:54.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-386547056649 2024-09-16T15:43:54.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:43:54.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T15:43:54.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-386547056649 2024-09-16T15:43:54.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:43:54.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=386547056649 2024-09-16T15:43:54.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 386547056649' 2024-09-16T15:43:54.355 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 386547056649 2024-09-16T15:43:54.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T15:43:54.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 386547056650 -lt 386547056649 2024-09-16T15:43:54.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:43:54.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:43:54.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:43:55.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-16T15:43:55.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:43:55.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:43:55.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:43:55.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:43:55.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:43:55.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:43:55.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:43:55.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-16T15:43:55.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:43:55.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:43:55.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:43:56.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-16T15:43:56.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:43:56.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:43:56.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-16T15:43:56.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:43:56.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:43:56.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-16T15:43:56.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:43:56.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:43:56.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:43:56.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:43:56.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:43:56.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:43:56.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:43:56.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:43:56.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:43:56.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:43:56.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:43:56.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:43:56.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:43:56.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:43:56.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:43:56.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:43:56.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:43:56.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:43:56.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-16T15:43:57.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:43:57.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:43:57.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:43:57.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96021 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:43:57.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:43:57.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:43:57.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:43:57.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:43:57.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:43:57.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:43:57.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:43:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:43:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:43:57.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:43:57.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:43:57.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:43:57.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:43:57.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:43:57.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:43:57.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:43:57.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-16T15:43:57.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:43:57.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:43:57.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:43:57.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96021 2024-09-16T15:43:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:43:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:43:57.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-16T15:43:57.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:43:57.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:43:57.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:43:57.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.96021 2024-09-16T15:43:57.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:43:57.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T15:43:57.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:43:57.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-16T15:43:57.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-09-16T15:43:57.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:43:57.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-09-16T15:43:57.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-09-16T15:43:57.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-09-16T15:43:57.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-09-16T15:43:57.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-09-16T15:43:57.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-09-16T15:43:57.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=d8dffaf9-3193-48af-a517-b2c8aae84901 --auth-supported=none ' 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-09-16T15:43:57.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:43:57.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:43:57.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:43:57.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:43:57.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-16T15:43:57.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-16T15:43:57.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:43:57.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:43:57.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:43:57.587 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:43:57.587 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:43:57.587 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:43:57.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:43:57.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:43:57.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:43:57.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:43:57.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:43:57.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:43:57.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:43:57.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:43:57.823 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:43:57.823 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:43:57.823 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:43:57.823 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:43:57.823 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:43:57.824 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:43:57.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96021/ceph-mon.a.asok 2024-09-16T15:43:57.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:43:57.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96021/ceph-mon.a.asok config get fsid 2024-09-16T15:43:57.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:43:57.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:43:57.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:43:57.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:43:57.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:43:57.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:43:57.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:43:57.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:43:57.947 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:43:57.947 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:43:57.947 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:43:57.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96021/ceph-mon.a.asok 2024-09-16T15:43:57.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:43:57.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96021/ceph-mon.a.asok config get mon_host 2024-09-16T15:43:58.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-09-16T15:43:58.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:43:58.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:43:58.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:43:58.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:43:58.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-16T15:43:58.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:43:59.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:43:59.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:43:59.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:43:59.002 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:43:59.003 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:43:59.003 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:43:59.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:43:59.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:43:59.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:43:59.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-09-16T15:43:59.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:43:59.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d8dffaf9-3193-48af-a517-b2c8aae84901 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:43:59.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:43:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:43:59.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:43:59.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:43:59.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:43:59.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-16T15:43:59.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:43:59.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=20a48944-f5fc-4b8e-9562-b860e3ebcd8c 2024-09-16T15:43:59.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 20a48944-f5fc-4b8e-9562-b860e3ebcd8c' 2024-09-16T15:43:59.076 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 20a48944-f5fc-4b8e-9562-b860e3ebcd8c 2024-09-16T15:43:59.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:43:59.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/UuhmOaoGBhAABFBamS1bCyUDqCGsae11bg== 2024-09-16T15:43:59.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/UuhmOaoGBhAABFBamS1bCyUDqCGsae11bg=="}' 2024-09-16T15:43:59.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 20a48944-f5fc-4b8e-9562-b860e3ebcd8c -i td/osd-bluefs-volume-ops/0/new.json 2024-09-16T15:43:59.480 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:43:59.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-16T15:43:59.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d8dffaf9-3193-48af-a517-b2c8aae84901 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/UuhmOaoGBhAABFBamS1bCyUDqCGsae11bg== --osd-uuid 20a48944-f5fc-4b8e-9562-b860e3ebcd8c 2024-09-16T15:43:59.537 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:59.537+0000 7ff96bf277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:59.539 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:59.537+0000 7ff96bf277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:59.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:59.541+0000 7ff96bf277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:43:59.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:59.541+0000 7ff96bf277c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-16T15:43:59.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:59.541+0000 7ff96bf277c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-16T15:43:59.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:59.541+0000 7ff96bf277c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-16T15:43:59.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:43:59.585+0000 7ff96bf277c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-16T15:44:07.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-16T15:44:07.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:44:07.755 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:44:07.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:44:07.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:44:08.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:44:08.329 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:44:08.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:44:08.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d8dffaf9-3193-48af-a517-b2c8aae84901 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:44:08.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:44:08.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:44:08.367 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:44:08.365+0000 7f78267707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:44:08.395 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:44:08.393+0000 7f78267707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:44:08.397 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:44:08.397+0000 7f78267707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:44:08.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:44:08.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:44:08.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:44:08.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:44:08.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:44:08.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:44:08.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:44:08.766 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:44:08.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:44:08.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:44:09.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:44:09.222 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:44:09.221+0000 7f78267707c0 -1 Falling back to public interface 2024-09-16T15:44:10.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:44:10.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:44:10.199 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:44:10.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:44:10.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:44:10.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:44:10.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:44:11.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:44:11.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:44:11.642 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:44:11.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:44:11.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:44:11.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:44:12.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:44:13.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:44:13.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:44:13.123 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:44:13.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:44:13.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:44:13.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:44:13.271 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:44:13.269+0000 7f78267707c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:44:13.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:44:14.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:44:14.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:44:14.606 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:44:14.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:44:14.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:44:14.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:44:16.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:44:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:44:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:44:17.038 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:44:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:44:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:44:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:44:17.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:44:18.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:44:18.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:44:18.791 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:44:18.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:44:18.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:44:18.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:44:19.216 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4199690102,v1:127.0.0.1:6803/4199690102] [v2:127.0.0.1:6804/4199690102,v1:127.0.0.1:6805/4199690102] exists,up 20a48944-f5fc-4b8e-9562-b860e3ebcd8c 2024-09-16T15:44:19.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:44:19.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:44:19.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:44:19.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-16T15:44:19.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=144492 2024-09-16T15:44:19.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-09-16T15:44:24.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-09-16T15:44:24.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-16T15:44:24.910 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T15:44:24.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:44:25.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-09-16T15:44:25.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-09-16T15:44:25.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-09-16T15:44:25.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-09-16T15:44:25.979 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-16T15:44:25.981 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-09-16T15:44:25.981 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_145775 2024-09-16T15:44:26.016 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:44:26.016 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-16T15:44:27.016 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 32 16 63.9923 64 0.979535 0.916119 2024-09-16T15:44:28.016 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 48 32 63.992 64 0.862444 0.896597 2024-09-16T15:44:29.016 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 64 48 63.9921 64 1.15678 0.97598 2024-09-16T15:44:30.017 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 65 49 48.9931 4 1.04433 0.977375 2024-09-16T15:44:31.017 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 81 65 51.9927 64 1.24559 0.995718 2024-09-16T15:44:32.017 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 97 81 53.9926 64 1.11901 1.05695 2024-09-16T15:44:33.017 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 113 97 55.4212 64 1.44563 1.12152 2024-09-16T15:44:34.017 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 129 113 56.4926 64 0.868512 1.07763 2024-09-16T15:44:35.017 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 144 128 56.8812 60 1.33114 1.10788 2024-09-16T15:44:36.017 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 160 144 57.5921 64 0.971002 1.09537 2024-09-16T15:44:37.018 INFO:tasks.workunit.client.0.smithi039.stdout: 11 15 160 145 52.7202 4 1.00227 1.09473 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 11.1997 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 160 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4194304 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4194304 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 57.1445 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 24.1209 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 64 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 4 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 14 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 6.03023 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 16 2024-09-16T15:44:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 1 2024-09-16T15:44:38.019 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 1.1127 2024-09-16T15:44:38.019 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.219634 2024-09-16T15:44:38.019 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 1.47005 2024-09-16T15:44:38.019 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.171119 2024-09-16T15:44:38.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-09-16T15:44:48.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-09-16T15:44:48.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-16T15:44:48.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1060110336 2024-09-16T15:44:48.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-16T15:44:48.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-09-16T15:44:48.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=55443456 2024-09-16T15:44:48.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-09-16T15:44:48.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 55443456 -eq 0 2024-09-16T15:44:48.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-09-16T15:44:48.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 55443456 -gt 0 2024-09-16T15:44:48.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 144492 2024-09-16T15:44:48.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-09-16T15:44:49.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 144492 2024-09-16T15:44:49.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (144492) - No such process 2024-09-16T15:44:49.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-09-16T15:44:49.948 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T15:44:49.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-09-16T15:44:49.999 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:44:59.537 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-16T15:44:59.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-09-16T15:44:59.590 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:45:01.373 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x100000000 : using 0x429f7000(1.0 GiB) 2024-09-16T15:45:01.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-09-16T15:45:03.449 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:45:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-09-16T15:45:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:45:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:45:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:45:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:45:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-16T15:45:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d8dffaf9-3193-48af-a517-b2c8aae84901 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-16T15:45:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:45:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:45:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:45:03.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-16T15:45:03.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-16T15:45:03.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:45:03.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:45:03.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:45:03.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:45:03.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:45:03.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:45:03.494 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:45:03.494 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:45:03.494 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:45:03.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:45:03.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:45:03.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:45:03.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:45:03.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:45:03.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:45:03.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:45:03.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:45:03.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:45:03.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:45:03.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:45:03.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-16T15:45:03.496 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:45:03.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:45:03.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d8dffaf9-3193-48af-a517-b2c8aae84901 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:45:03.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-16T15:45:03.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:45:03.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:45:03.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:45:03.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:45:03.540 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:03.536+0000 7f489edd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:45:03.542 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:03.540+0000 7f489edd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:45:03.544 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:03.540+0000 7f489edd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:45:03.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:45:03.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:45:03.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:45:03.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:45:03.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:45:03.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:03.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:45:03.925 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:45:03.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:03.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:04.126 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:04.124+0000 7f489edd77c0 -1 Falling back to public interface 2024-09-16T15:45:04.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:05.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:05.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:05.375 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:45:05.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:45:05.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:05.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:05.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:06.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:06.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:06.813 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:45:06.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:45:06.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:06.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:07.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:08.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:08.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:08.257 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:45:08.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:45:08.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:08.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:08.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:09.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:09.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:09.700 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:45:09.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:45:09.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:09.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:10.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:11.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:11.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:11.129 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:45:11.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:45:11.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:11.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:11.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:11.941 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:11.940+0000 7f489edd77c0 -1 osd.0 9 log_to_monitors true 2024-09-16T15:45:12.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:12.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:12.562 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:45:12.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:45:12.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:12.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:13.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:14.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:14.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:14.536 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:45:14.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:45:14.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:14.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:15.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:16.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:16.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:16.739 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:45:16.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:45:16.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:16.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:17.381 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 12 up_thru 12 down_at 10 last_clean_interval [5,9) [v2:127.0.0.1:6802/774065916,v1:127.0.0.1:6803/774065916] [v2:127.0.0.1:6804/774065916,v1:127.0.0.1:6805/774065916] exists,up 20a48944-f5fc-4b8e-9562-b860e3ebcd8c 2024-09-16T15:45:17.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:45:17.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:45:17.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:45:17.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-16T15:45:17.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=147003 2024-09-16T15:45:17.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-09-16T15:45:17.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:45:17.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:45:17.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:45:17.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:45:17.385 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:45:17.385 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:45:17.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:45:17.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:45:17.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:45:17.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:45:17.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:45:17.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:45:17.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:45:17.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:45:17.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:45:17.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T15:45:17.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:45:17.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:45:17.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:45:18.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607556 2024-09-16T15:45:18.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607556 2024-09-16T15:45:18.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-51539607556' 2024-09-16T15:45:18.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:45:18.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-51539607556 2024-09-16T15:45:18.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:45:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:45:18.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-51539607556 2024-09-16T15:45:18.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:45:18.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607556 2024-09-16T15:45:18.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 51539607556' 2024-09-16T15:45:18.106 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 51539607556 2024-09-16T15:45:18.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:45:18.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607554 -lt 51539607556 2024-09-16T15:45:18.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:45:19.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:45:19.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:45:19.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607556 -lt 51539607556 2024-09-16T15:45:19.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:45:19.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:45:19.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:45:20.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-16T15:45:20.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:45:20.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:45:20.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:45:20.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:45:20.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:45:20.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:45:20.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:45:20.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-16T15:45:20.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:45:20.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:45:20.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:45:21.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-16T15:45:21.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:45:21.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:45:21.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-16T15:45:21.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:45:21.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:45:21.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-16T15:45:21.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:45:21.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:45:21.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:45:21.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:45:21.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:45:21.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:45:21.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:45:21.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:45:21.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:45:21.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:45:21.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:45:21.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:45:21.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:45:21.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:45:21.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:45:21.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:45:21.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:45:21.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:45:21.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-16T15:45:21.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:45:21.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:45:21.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:45:21.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96021 2024-09-16T15:45:21.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:45:21.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:45:21.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:45:21.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:45:21.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:45:21.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:45:21.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:45:21.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:45:21.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:45:21.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:45:21.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:45:21.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:45:21.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:45:21.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:45:21.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:45:21.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:45:21.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-16T15:45:21.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:45:21.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:45:21.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:45:21.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96021 2024-09-16T15:45:21.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:45:21.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:45:21.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-16T15:45:21.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:45:21.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:45:21.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:45:21.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.96021 2024-09-16T15:45:21.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:45:21.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T15:45:21.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:45:21.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-16T15:45:21.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-09-16T15:45:21.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:45:21.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-09-16T15:45:21.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-09-16T15:45:21.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-09-16T15:45:21.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-09-16T15:45:21.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-09-16T15:45:21.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-09-16T15:45:21.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-09-16T15:45:21.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=baf6250d-d41f-4dd7-a925-764f206f6e5e --auth-supported=none ' 2024-09-16T15:45:21.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-16T15:45:21.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-16T15:45:22.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:45:22.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:45:22.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:45:22.148 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:45:22.148 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:45:22.148 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:45:22.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:45:22.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:45:22.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:45:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:45:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:45:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:45:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:45:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:45:22.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:45:22.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:45:22.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:45:22.372 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:45:22.372 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:45:22.372 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:45:22.372 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96021/ceph-mon.a.asok 2024-09-16T15:45:22.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:45:22.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96021/ceph-mon.a.asok config get fsid 2024-09-16T15:45:22.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:45:22.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:45:22.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:45:22.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:45:22.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:45:22.493 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:45:22.493 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:45:22.493 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:45:22.493 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:45:22.493 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:45:22.493 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:45:22.494 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96021/ceph-mon.a.asok 2024-09-16T15:45:22.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:45:22.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96021/ceph-mon.a.asok config get mon_host 2024-09-16T15:45:22.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-09-16T15:45:22.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:45:22.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:45:22.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:45:22.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:45:22.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-16T15:45:22.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:45:23.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:45:23.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:45:23.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:45:23.150 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:45:23.151 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:45:23.151 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:45:23.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:45:23.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:45:23.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:45:23.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-09-16T15:45:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:45:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:45:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:45:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:45:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-16T15:45:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d8dffaf9-3193-48af-a517-b2c8aae84901 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=baf6250d-d41f-4dd7-a925-764f206f6e5e --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-16T15:45:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:45:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:45:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:45:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-16T15:45:23.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:45:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:45:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:45:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:45:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:45:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:45:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:45:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:45:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:45:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:45:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-16T15:45:23.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:45:23.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5d1aeeae-5b63-49c0-9346-cb52a4355636 2024-09-16T15:45:23.220 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 5d1aeeae-5b63-49c0-9346-cb52a4355636 2024-09-16T15:45:23.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5d1aeeae-5b63-49c0-9346-cb52a4355636' 2024-09-16T15:45:23.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:45:23.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTUuhmn/TJDhAAoqAqtpdagXQuj5eyiaZHcA== 2024-09-16T15:45:23.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTUuhmn/TJDhAAoqAqtpdagXQuj5eyiaZHcA=="}' 2024-09-16T15:45:23.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5d1aeeae-5b63-49c0-9346-cb52a4355636 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-16T15:45:23.632 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:45:23.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-16T15:45:23.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d8dffaf9-3193-48af-a517-b2c8aae84901 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=baf6250d-d41f-4dd7-a925-764f206f6e5e --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTUuhmn/TJDhAAoqAqtpdagXQuj5eyiaZHcA== --osd-uuid 5d1aeeae-5b63-49c0-9346-cb52a4355636 2024-09-16T15:45:23.699 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:23.696+0000 7ff9e0a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:45:23.714 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:23.712+0000 7ff9e0a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:45:23.715 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:23.712+0000 7ff9e0a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:45:23.715 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:23.712+0000 7ff9e0a897c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-16T15:45:23.716 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:23.712+0000 7ff9e0a897c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-16T15:45:23.716 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:23.712+0000 7ff9e0a897c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-16T15:45:23.806 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:23.804+0000 7ff9e0a897c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-16T15:45:31.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-16T15:45:31.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:45:31.495 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:45:31.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:45:31.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:45:32.066 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:45:32.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:45:32.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:45:32.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d8dffaf9-3193-48af-a517-b2c8aae84901 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=baf6250d-d41f-4dd7-a925-764f206f6e5e --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:45:32.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:45:32.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:45:32.111 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:32.107+0000 7f05d4a6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:45:32.114 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:32.111+0000 7f05d4a6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:45:32.115 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:32.111+0000 7f05d4a6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:45:32.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:45:32.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:45:32.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:45:32.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:45:32.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:45:32.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:32.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:45:32.526 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:45:32.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:32.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:32.903 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:32.899+0000 7f05d4a6a7c0 -1 Falling back to public interface 2024-09-16T15:45:33.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:34.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:34.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:34.013 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:45:34.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:45:34.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:34.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:34.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:35.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:35.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:35.449 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:45:35.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:45:35.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:35.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:35.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:36.876 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:45:36.871+0000 7f05d4a6a7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:45:36.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:36.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:36.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:45:36.883 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:45:36.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:36.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:37.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:38.574 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:45:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:45:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:38.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:39.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:40.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:40.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:40.815 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:45:40.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:45:40.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:40.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:41.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:45:42.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:45:42.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:45:42.307 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:45:42.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:45:42.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:45:42.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:45:42.747 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3189144921,v1:127.0.0.1:6803/3189144921] [v2:127.0.0.1:6804/3189144921,v1:127.0.0.1:6805/3189144921] exists,up 5d1aeeae-5b63-49c0-9346-cb52a4355636 2024-09-16T15:45:42.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:45:42.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:45:42.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:45:42.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-16T15:45:42.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=149746 2024-09-16T15:45:42.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-09-16T15:45:47.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-09-16T15:45:47.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-16T15:45:48.472 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T15:45:48.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:45:49.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-16T15:45:49.542 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-16T15:45:49.542 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-16T15:45:49.542 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_150772 2024-09-16T15:45:49.542 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:45:49.542 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-16T15:45:50.542 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 144 128 0.499952 0.5 0.151154 0.123772 2024-09-16T15:45:51.542 INFO:tasks.workunit.client.0.smithi039.stdout: 2 15 294 279 0.544867 0.589844 0.0921942 0.111519 2024-09-16T15:45:52.542 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 464 448 0.58327 0.660156 0.118076 0.106044 2024-09-16T15:45:53.543 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 625 609 0.594656 0.628906 0.0837366 0.103007 2024-09-16T15:45:54.543 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 768 752 0.587432 0.558594 0.0926814 0.105936 2024-09-16T15:45:55.543 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 929 913 0.594327 0.628906 0.092082 0.104225 2024-09-16T15:45:56.543 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 1072 1056 0.589209 0.558594 0.101095 0.105258 2024-09-16T15:45:57.543 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 1248 1232 0.601483 0.6875 0.0838342 0.103385 2024-09-16T15:45:58.544 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 1393 1377 0.597576 0.566406 0.0837517 0.104143 2024-09-16T15:45:59.544 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 1521 1505 0.587811 0.5 0.0920156 0.105513 2024-09-16T15:46:00.544 INFO:tasks.workunit.client.0.smithi039.stdout: 11 16 1665 1649 0.585505 0.5625 0.0838132 0.106298 2024-09-16T15:46:01.544 INFO:tasks.workunit.client.0.smithi039.stdout: 12 16 1825 1809 0.588788 0.625 0.108883 0.105349 2024-09-16T15:46:02.544 INFO:tasks.workunit.client.0.smithi039.stdout: 13 16 2000 1984 0.596073 0.683594 0.0926408 0.104718 2024-09-16T15:46:03.544 INFO:tasks.workunit.client.0.smithi039.stdout: 14 16 2161 2145 0.598411 0.628906 0.0837796 0.103959 2024-09-16T15:46:04.544 INFO:tasks.workunit.client.0.smithi039.stdout: 15 16 2321 2305 0.600178 0.625 0.134051 0.103724 2024-09-16T15:46:05.545 INFO:tasks.workunit.client.0.smithi039.stdout: 16 16 2465 2449 0.597817 0.5625 0.117169 0.103921 2024-09-16T15:46:06.545 INFO:tasks.workunit.client.0.smithi039.stdout: 17 16 2625 2609 0.59941 0.625 0.117623 0.103908 2024-09-16T15:46:07.545 INFO:tasks.workunit.client.0.smithi039.stdout: 18 16 2784 2768 0.600609 0.621094 0.101974 0.103863 2024-09-16T15:46:08.545 INFO:tasks.workunit.client.0.smithi039.stdout: 19 16 2929 2913 0.598805 0.566406 0.0836754 0.10415 2024-09-16T15:46:09.545 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:46:09.547804+0000 min lat: 0.0481569 max lat: 0.275361 avg lat: 0.103374 2024-09-16T15:46:09.545 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:46:09.545 INFO:tasks.workunit.client.0.smithi039.stdout: 20 16 3105 3089 0.603234 0.6875 0.0670625 0.103374 2024-09-16T15:46:10.545 INFO:tasks.workunit.client.0.smithi039.stdout: 21 16 3233 3217 0.598313 0.5 0.143185 0.103979 2024-09-16T15:46:11.545 INFO:tasks.workunit.client.0.smithi039.stdout: 22 16 3408 3392 0.602185 0.683594 0.0928816 0.103635 2024-09-16T15:46:12.545 INFO:tasks.workunit.client.0.smithi039.stdout: 23 16 3552 3536 0.600457 0.5625 0.0921781 0.10404 2024-09-16T15:46:13.546 INFO:tasks.workunit.client.0.smithi039.stdout: 24 16 3713 3697 0.601638 0.628906 0.0838189 0.103592 2024-09-16T15:46:14.546 INFO:tasks.workunit.client.0.smithi039.stdout: 25 16 3872 3856 0.602412 0.621094 0.0923993 0.103605 2024-09-16T15:46:15.546 INFO:tasks.workunit.client.0.smithi039.stdout: 26 16 4017 4001 0.601024 0.566406 0.102112 0.103629 2024-09-16T15:46:16.546 INFO:tasks.workunit.client.0.smithi039.stdout: 27 16 4161 4145 0.599594 0.5625 0.16766 0.103946 2024-09-16T15:46:17.546 INFO:tasks.workunit.client.0.smithi039.stdout: 28 16 4321 4305 0.600498 0.625 0.0921541 0.103776 2024-09-16T15:46:18.546 INFO:tasks.workunit.client.0.smithi039.stdout: 29 16 4465 4449 0.599184 0.5625 0.120091 0.104152 2024-09-16T15:46:19.546 INFO:tasks.workunit.client.0.smithi039.stdout: 30 16 4625 4609 0.600041 0.625 0.117223 0.103884 2024-09-16T15:46:20.547 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 30.0816 2024-09-16T15:46:20.547 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 4625 2024-09-16T15:46:20.547 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4096 2024-09-16T15:46:20.547 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4096 2024-09-16T15:46:20.547 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.600579 2024-09-16T15:46:20.547 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.0538877 2024-09-16T15:46:20.547 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.6875 2024-09-16T15:46:20.547 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.5 2024-09-16T15:46:20.547 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 153 2024-09-16T15:46:20.547 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 13.7953 2024-09-16T15:46:20.548 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 176 2024-09-16T15:46:20.548 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 128 2024-09-16T15:46:20.548 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.103899 2024-09-16T15:46:20.548 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.023036 2024-09-16T15:46:20.548 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 0.275361 2024-09-16T15:46:20.548 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.0481569 2024-09-16T15:46:20.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-09-16T15:46:25.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-16T15:46:25.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-16T15:46:25.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-09-16T15:46:25.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-16T15:46:25.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-16T15:46:25.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-09-16T15:46:25.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-09-16T15:46:25.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-09-16T15:46:25.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-16T15:46:25.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4275920896 2024-09-16T15:46:25.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 149746 2024-09-16T15:46:25.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-09-16T15:46:26.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 149746 2024-09-16T15:46:26.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (149746) - No such process 2024-09-16T15:46:26.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-09-16T15:46:27.448 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T15:46:27.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-09-16T15:46:27.502 INFO:tasks.workunit.client.0.smithi039.stdout:allocmap bluestore.allocmap 2024-09-16T15:46:30.540 INFO:tasks.workunit.client.0.smithi039.stdout:allocmap success 2024-09-16T15:46:30.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:446: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-09-16T15:46:33.620 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:46:33.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:448: TEST_bluestore_expand: requested_space=4294967296 2024-09-16T15:46:33.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:449: TEST_bluestore_expand: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-16T15:46:33.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:450: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-16T15:46:33.679 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:46:35.951 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0x2300000(35 MiB) 2024-09-16T15:46:35.951 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0x100000000 : using 0x122a000(18 MiB) 2024-09-16T15:46:35.952 INFO:tasks.workunit.client.0.smithi039.stdout:Expanding DB/WAL... 2024-09-16T15:46:35.952 INFO:tasks.workunit.client.0.smithi039.stdout:1 : expanding to 0x1073741824 2024-09-16T15:46:35.980 INFO:tasks.workunit.client.0.smithi039.stdout:1 : size label updated to 1073741824 2024-09-16T15:46:36.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:453: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-09-16T15:46:39.792 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-16T15:46:39.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:456: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-09-16T15:46:39.843 INFO:tasks.workunit.client.0.smithi039.stdout:qfsck bluestore.quick-fsck 2024-09-16T15:46:42.637 INFO:tasks.workunit.client.0.smithi039.stdout:qfsck success 2024-09-16T15:46:42.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:458: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-16T15:46:42.684 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-16T15:46:45.466 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0x2300000(35 MiB) 2024-09-16T15:46:45.466 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0x100000000 : using 0x122a000(18 MiB) 2024-09-16T15:46:45.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:460: TEST_bluestore_expand: activate_osd td/osd-bluefs-volume-ops 0 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d8dffaf9-3193-48af-a517-b2c8aae84901 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=baf6250d-d41f-4dd7-a925-764f206f6e5e --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:46:45.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:46:45.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-16T15:46:45.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:46:45.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:46:45.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:46:45.477 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:46:45.477 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:46:45.477 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:46:45.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:46:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' 2024-09-16T15:46:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:46:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-16T15:46:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-16T15:46:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:46:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:46:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:46:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:46:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:46:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:46:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-16T15:46:45.479 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:46:45.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:46:45.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=fee3b25f-f6cc-4bcb-9051-f167c20b2a37 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d8dffaf9-3193-48af-a517-b2c8aae84901 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=baf6250d-d41f-4dd7-a925-764f206f6e5e --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96021/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:46:45.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-16T15:46:45.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:46:45.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:46:45.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:46:45.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:46:45.533 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:46:45.534+0000 7fde28c0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:46:45.536 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:46:45.534+0000 7fde28c0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:46:45.538 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:46:45.538+0000 7fde28c0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:46:45.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:46:45.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:46:45.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:46:45.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:46:45.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:46:45.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:46:45.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:46:45.939 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:46:45.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:46:45.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:46:46.323 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:46:46.322+0000 7fde28c0a7c0 -1 Falling back to public interface 2024-09-16T15:46:46.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:46:47.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:46:47.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:46:47.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:46:47.396 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:46:47.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:46:47.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:46:47.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:46:48.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:46:48.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:46:48.875 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:46:48.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:46:48.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:46:48.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:46:49.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:46:50.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:46:50.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:46:50.306 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:46:50.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:46:50.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:46:50.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:46:50.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:46:51.330 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:46:51.330+0000 7fde28c0a7c0 -1 osd.0 10 log_to_monitors true 2024-09-16T15:46:51.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:46:51.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:46:51.778 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:46:51.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:46:51.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:46:51.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:46:52.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:46:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:46:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:46:53.472 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:46:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:46:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:46:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:46:53.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:46:54.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:46:54.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:46:54.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:46:54.936 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:46:54.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:46:54.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:46:55.391 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/50178507,v1:127.0.0.1:6803/50178507] [v2:127.0.0.1:6804/50178507,v1:127.0.0.1:6805/50178507] exists,up 5d1aeeae-5b63-49c0-9346-cb52a4355636 2024-09-16T15:46:55.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:46:55.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:46:55.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:46:55.392 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-16T15:46:55.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: osd_pid0=153123 2024-09-16T15:46:55.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:463: TEST_bluestore_expand: wait_for_clean 2024-09-16T15:46:55.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:46:55.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:46:55.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:46:55.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:46:55.394 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:46:55.394 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:46:55.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:46:55.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:46:55.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:46:55.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:46:55.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:46:55.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:46:55.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:46:55.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:46:55.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:46:55.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T15:46:55.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:46:55.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:46:55.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:46:56.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574851 2024-09-16T15:46:56.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574851 2024-09-16T15:46:56.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-55834574851' 2024-09-16T15:46:56.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:46:56.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-55834574851 2024-09-16T15:46:56.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:46:56.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:46:56.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-55834574851 2024-09-16T15:46:56.206 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:46:56.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574851 2024-09-16T15:46:56.207 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 55834574851 2024-09-16T15:46:56.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 55834574851' 2024-09-16T15:46:56.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:46:56.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 55834574851 2024-09-16T15:46:56.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:46:57.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:46:57.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:46:58.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574852 -lt 55834574851 2024-09-16T15:46:58.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:46:58.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:46:58.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:46:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-16T15:46:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:46:58.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:46:58.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:46:58.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:46:58.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:46:58.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:46:58.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:46:59.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-16T15:46:59.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:46:59.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:46:59.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:46:59.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-16T15:46:59.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:46:59.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:46:59.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-16T15:46:59.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-16T15:46:59.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: total_space_after=4294967296 2024-09-16T15:46:59.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-16T15:46:59.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep free 2024-09-16T15:46:59.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d: -f 2 2024-09-16T15:46:59.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-16T15:46:59.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d, -f 1 2024-09-16T15:46:59.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-16T15:47:00.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: free_space_after=4275879936 2024-09-16T15:47:00.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:468: TEST_bluestore_expand: '[4294967296' '!=' '4294967296]' 2024-09-16T15:47:00.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 468: [4294967296: command not found 2024-09-16T15:47:00.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:474: TEST_bluestore_expand: total_space_added=0 2024-09-16T15:47:00.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:475: TEST_bluestore_expand: free_space_added=-40960 2024-09-16T15:47:00.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:477: TEST_bluestore_expand: let 'new_used_space=(0 - -40960)' 2024-09-16T15:47:00.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:478: TEST_bluestore_expand: echo 40960 2024-09-16T15:47:00.140 INFO:tasks.workunit.client.0.smithi039.stdout:40960 2024-09-16T15:47:00.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:480: TEST_bluestore_expand: '[' 40960 -gt 131072 ']' 2024-09-16T15:47:00.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 153123 2024-09-16T15:47:00.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: sleep 1 2024-09-16T15:47:01.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 153123 2024-09-16T15:47:01.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 487: kill: (153123) - No such process 2024-09-16T15:47:01.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:488: TEST_bluestore_expand: ceph osd down 0 2024-09-16T15:47:01.765 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T15:47:01.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:490: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-09-16T15:47:01.826 INFO:tasks.workunit.client.0.smithi039.stdout:qfsck bluestore.quick-fsck 2024-09-16T15:47:04.365 INFO:tasks.workunit.client.0.smithi039.stdout:qfsck success 2024-09-16T15:47:04.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-16T15:47:04.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:47:04.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:47:04.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-16T15:47:04.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:47:04.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:47:04.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:47:04.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:47:04.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:47:04.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:47:04.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:47:04.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:47:04.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:47:04.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:47:04.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:47:04.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:47:04.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:47:04.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:47:04.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:47:04.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:47:04.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:47:04.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:47:04.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-16T15:47:04.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:47:04.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:04.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:47:04.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96021 2024-09-16T15:47:04.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:47:04.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:47:04.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T15:47:04.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-bluefs-volume-ops 0 2024-09-16T15:47:04.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-16T15:47:04.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T15:47:04.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-16T15:47:04.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:47:04.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:47:04.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:47:04.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:47:04.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:47:04.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:47:04.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:47:04.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:47:04.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:47:04.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:47:04.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:47:04.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:47:04.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:47:04.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:47:04.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:47:04.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:47:04.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:47:04.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T15:47:04.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-16T15:47:04.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:47:04.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:04.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96021 2024-09-16T15:47:04.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96021 2024-09-16T15:47:04.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:47:04.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:47:04.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T15:47:04.595 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T15:47:04.595 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T15:47:04.606 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2024-09-16T15:47:04.607 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2024-09-16T15:47:04.657 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T15:47:04.662 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T15:47:04.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-config 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2024-09-16T15:47:04.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2024-09-16T15:47:04.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=cc0b42e1-c1d1-4359-bd8f-ff5fe11fb7ec --auth-supported=none ' 2024-09-16T15:47:04.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-09-16T15:47:04.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2024-09-16T15:47:04.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_config_track 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_default_adjustment' 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:47:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:47:04.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:47:04.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:47:04.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:47:04.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:47:04.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:47:04.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:47:04.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:47:04.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:47:04.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:47:04.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:47:04.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:47:04.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:47:04.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:47:04.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:47:04.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:47:04.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-16T15:47:04.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:47:04.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:04.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:04.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154844 2024-09-16T15:47:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:47:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:47:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-09-16T15:47:04.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:47:04.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:04.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:04.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.154844 2024-09-16T15:47:04.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:47:04.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T15:47:04.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T15:47:04.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:47:04.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-09-16T15:47:04.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2024-09-16T15:47:04.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2024-09-16T15:47:04.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2024-09-16T15:47:04.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-09-16T15:47:04.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:47:04.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:47:04.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:47:04.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-09-16T15:47:04.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-09-16T15:47:05.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:47:05.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:47:05.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:47:05.062 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:47:05.062 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:05.062 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:05.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154844/$cluster-$name.asok' 2024-09-16T15:47:05.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154844/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:47:05.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:47:05.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:47:05.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:47:05.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:47:05.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:47:05.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:47:05.293 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:47:05.293 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:47:05.293 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:47:05.294 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:47:05.294 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:05.294 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:05.294 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154844/ceph-mon.a.asok 2024-09-16T15:47:05.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:47:05.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154844/ceph-mon.a.asok config get fsid 2024-09-16T15:47:05.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:47:05.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:47:05.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:47:05.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:47:05.413 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:47:05.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:47:05.413 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:47:05.413 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:47:05.414 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:47:05.414 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:05.414 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:05.414 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154844/ceph-mon.a.asok 2024-09-16T15:47:05.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:47:05.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154844/ceph-mon.a.asok config get mon_host 2024-09-16T15:47:05.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2024-09-16T15:47:05.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-09-16T15:47:05.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:47:05.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:47:05.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:47:05.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-09-16T15:47:05.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:47:06.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:47:06.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:47:06.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:47:06.422 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:47:06.422 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:06.422 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:06.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154844/$cluster-$name.asok' 2024-09-16T15:47:06.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:47:06.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154844/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:47:06.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2024-09-16T15:47:06.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2024-09-16T15:47:06.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-09-16T15:47:06.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cc0b42e1-c1d1-4359-bd8f-ff5fe11fb7ec --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-09-16T15:47:06.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154844/$cluster-$name.asok' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.154844/$cluster-$name.asok' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:47:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:47:06.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:47:06.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2024-09-16T15:47:06.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-09-16T15:47:06.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:47:06.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3057b371-6ac7-427a-a5de-350163a91825 2024-09-16T15:47:06.462 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 3057b371-6ac7-427a-a5de-350163a91825 2024-09-16T15:47:06.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3057b371-6ac7-427a-a5de-350163a91825' 2024-09-16T15:47:06.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:47:06.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6Uuhmc+6BHBAAzlSNSK60X4x9UBg3YgLJvA== 2024-09-16T15:47:06.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6Uuhmc+6BHBAAzlSNSK60X4x9UBg3YgLJvA=="}' 2024-09-16T15:47:06.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3057b371-6ac7-427a-a5de-350163a91825 -i td/osd-config/0/new.json 2024-09-16T15:47:06.820 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:47:06.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-09-16T15:47:06.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cc0b42e1-c1d1-4359-bd8f-ff5fe11fb7ec --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQD6Uuhmc+6BHBAAzlSNSK60X4x9UBg3YgLJvA== --osd-uuid 3057b371-6ac7-427a-a5de-350163a91825 2024-09-16T15:47:06.872 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:06.874+0000 7f099c14a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:06.874 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:06.874+0000 7f099c14a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:06.875 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:06.874+0000 7f099c14a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:06.876 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:06.878+0000 7f099c14a7c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-16T15:47:06.876 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:06.878+0000 7f099c14a7c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-16T15:47:06.876 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:06.878+0000 7f099c14a7c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-16T15:47:06.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:06.962+0000 7f099c14a7c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-09-16T15:47:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-09-16T15:47:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:47:12.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:47:12.815 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:47:12.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:47:13.388 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:47:13.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:47:13.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cc0b42e1-c1d1-4359-bd8f-ff5fe11fb7ec --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-09-16T15:47:13.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:47:13.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:47:13.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:47:13.421 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:13.422+0000 7f62cd8627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:13.425 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:13.426+0000 7f62cd8627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:13.426 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:13.426+0000 7f62cd8627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:13.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:47:13.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:47:13.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:47:13.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:47:13.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:47:13.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:13.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:47:13.838 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:47:13.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:13.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:14.009 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:14.010+0000 7f62cd8627c0 -1 Falling back to public interface 2024-09-16T15:47:14.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:15.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:15.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:15.333 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:47:15.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:47:15.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:15.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:15.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:16.442 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:16.441+0000 7f62cd8627c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:47:16.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:16.780 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:47:16.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:16.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:47:16.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:16.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:17.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:18.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:18.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:18.450 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:47:18.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:47:18.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:18.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:18.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:19.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:19.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:19.870 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:47:19.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:47:19.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:19.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:20.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:21.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:21.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:21.471 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:47:21.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:47:21.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:21.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:21.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:22.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:22.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:22.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:47:22.904 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:47:22.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:22.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:23.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:24.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:24.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:24.435 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:47:24.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:47:24.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:24.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:24.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:25.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:25.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:25.864 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:47:25.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:47:25.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:25.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:26.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:27.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:27.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:27.393 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:47:27.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:47:27.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:27.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:27.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:28.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:28.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:28.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:47:28.817 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:47:28.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:28.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:29.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:30.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:30.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:30.685 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:47:30.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:47:30.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:30.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:31.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:32.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:32.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:32.172 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:47:32.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:47:32.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:32.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:32.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:33.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:33.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:33.871 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:47:33.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:47:33.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:33.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:34.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:35.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:35.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:35.917 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:47:35.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:47:35.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:35.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:36.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:37.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:37.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:37.957 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:47:37.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:47:37.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:37.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:38.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:39.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:39.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:39.767 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:47:39.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:47:39.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:39.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:40.205 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2678783830,v1:127.0.0.1:6803/2678783830] [v2:127.0.0.1:6804/2678783830,v1:127.0.0.1:6805/2678783830] exists,up 3057b371-6ac7-427a-a5de-350163a91825 2024-09-16T15:47:40.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:47:40.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:47:40.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:47:40.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2024-09-16T15:47:40.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T15:47:40.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T15:47:40.208 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:47:40.208 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:40.208 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:40.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154844/ceph-osd.0.asok 2024-09-16T15:47:40.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2024-09-16T15:47:40.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.154844/ceph-osd.0.asok log flush 2024-09-16T15:47:40.310 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T15:47:40.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-09-16T15:47:40.324 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:47:16.741+0000 7f62cd8627c0 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-16T15:47:40.324 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:47:19.441+0000 7f62bd805640 10 log_client will send 2024-09-16T15:47:16.747010+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-16T15:47:40.324 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:47:19.777+0000 7f62be807640 10 log_client logged 2024-09-16T15:47:16.747010+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-16T15:47:40.324 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:47:38.813+0000 7f62c901c640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-16T15:47:40.324 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:47:39.445+0000 7f62bd805640 10 log_client will send 2024-09-16T15:47:38.816544+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-16T15:47:40.324 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:47:39.637+0000 7f62be807640 10 log_client logged 2024-09-16T15:47:38.816544+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-16T15:47:40.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-09-16T15:47:40.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-16T15:47:40.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:47:40.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-16T15:47:40.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:47:40.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:47:40.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:47:40.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:47:40.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:47:40.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:47:40.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:47:40.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:47:40.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:47:40.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:47:40.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:47:40.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:47:40.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:47:40.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:47:40.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:47:40.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:47:40.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:47:40.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:47:40.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-16T15:47:40.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:47:40.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:40.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:40.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154844 2024-09-16T15:47:40.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:47:40.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:47:40.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-09-16T15:47:40.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-09-16T15:47:40.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-09-16T15:47:40.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-09-16T15:47:40.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-16T15:47:40.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:47:40.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-16T15:47:40.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:47:40.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:47:40.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:47:40.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:47:40.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:47:40.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:47:40.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:47:40.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:47:40.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:47:40.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:47:40.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:47:40.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:47:40.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:47:40.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:47:40.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:47:40.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:47:40.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:47:40.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:47:40.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-16T15:47:40.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:47:40.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:40.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:40.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154844 2024-09-16T15:47:40.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:47:40.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:47:40.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-09-16T15:47:40.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:47:40.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:40.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:40.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.154844 2024-09-16T15:47:40.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-09-16T15:47:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-09-16T15:47:40.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:47:40.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:47:40.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:47:40.816 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:47:40.816 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:40.816 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:40.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154844/$cluster-$name.asok' 2024-09-16T15:47:40.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154844/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:47:41.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:47:41.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:47:41.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:47:41.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:47:41.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:47:41.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:47:41.021 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:47:41.021 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:47:41.021 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:47:41.022 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:47:41.022 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:41.022 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:41.022 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154844/ceph-mon.a.asok 2024-09-16T15:47:41.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:47:41.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154844/ceph-mon.a.asok config get fsid 2024-09-16T15:47:41.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:47:41.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:47:41.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:47:41.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:47:41.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:47:41.139 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:47:41.139 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:47:41.139 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:47:41.139 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:47:41.139 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:41.139 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:41.140 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154844/ceph-mon.a.asok 2024-09-16T15:47:41.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:47:41.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154844/ceph-mon.a.asok config get mon_host 2024-09-16T15:47:41.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2024-09-16T15:47:41.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-09-16T15:47:41.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:47:41.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:47:41.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:47:41.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-09-16T15:47:41.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:47:42.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:47:42.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:47:42.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:47:42.145 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:47:42.145 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:42.145 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:42.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154844/$cluster-$name.asok' 2024-09-16T15:47:42.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:47:42.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154844/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:47:42.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2024-09-16T15:47:42.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-09-16T15:47:42.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:47:42.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cc0b42e1-c1d1-4359-bd8f-ff5fe11fb7ec --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:47:42.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:47:42.211 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:47:42.211 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:47:42.211 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154844/$cluster-$name.asok' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.154844/$cluster-$name.asok' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:47:42.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:47:42.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-09-16T15:47:42.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:47:42.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dcc693c4-02b2-42ea-881d-5a8f6679dd56 2024-09-16T15:47:42.213 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 dcc693c4-02b2-42ea-881d-5a8f6679dd56 2024-09-16T15:47:42.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dcc693c4-02b2-42ea-881d-5a8f6679dd56' 2024-09-16T15:47:42.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:47:42.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAeU+hmDj8fDhAAU+y+i87XKezk4cdEDJwtMg== 2024-09-16T15:47:42.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAeU+hmDj8fDhAAU+y+i87XKezk4cdEDJwtMg=="}' 2024-09-16T15:47:42.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dcc693c4-02b2-42ea-881d-5a8f6679dd56 -i td/osd-config/0/new.json 2024-09-16T15:47:42.566 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:47:42.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-09-16T15:47:42.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cc0b42e1-c1d1-4359-bd8f-ff5fe11fb7ec --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAeU+hmDj8fDhAAU+y+i87XKezk4cdEDJwtMg== --osd-uuid dcc693c4-02b2-42ea-881d-5a8f6679dd56 2024-09-16T15:47:42.611 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:42.609+0000 7f10bf1f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:42.613 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:42.613+0000 7f10bf1f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:42.615 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:42.613+0000 7f10bf1f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:42.615 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:42.613+0000 7f10bf1f27c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-16T15:47:42.615 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:42.613+0000 7f10bf1f27c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-16T15:47:42.615 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:42.613+0000 7f10bf1f27c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-16T15:47:42.698 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:42.697+0000 7f10bf1f27c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-09-16T15:47:48.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-09-16T15:47:48.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:47:48.495 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:47:48.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:47:48.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:47:49.062 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:47:49.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:47:49.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:47:49.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cc0b42e1-c1d1-4359-bd8f-ff5fe11fb7ec --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:47:49.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:47:49.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:47:49.094 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:49.093+0000 7f0f96f9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:49.105+0000 7f0f96f9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:49.106 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:49.105+0000 7f0f96f9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:47:49.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:47:49.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:47:49.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:47:49.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:47:49.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:47:49.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:49.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:47:49.483 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:47:49.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:49.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:49.682 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:49.681+0000 7f0f96f9b7c0 -1 Falling back to public interface 2024-09-16T15:47:49.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:50.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:50.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:50.947 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:47:50.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:47:50.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:50.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:51.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:52.213 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:47:52.213+0000 7f0f96f9b7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:47:52.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:52.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:52.414 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:47:52.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:47:52.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:52.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:52.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:53.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:53.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:53.905 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:47:53.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:47:53.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:53.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:54.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:55.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:55.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:55.347 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:47:55.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:47:55.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:55.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:55.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:56.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:56.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:56.925 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:47:56.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:47:56.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:56.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:57.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:58.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:58.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:58.394 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:47:58.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:47:58.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:58.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:47:58.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:47:59.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:47:59.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:47:59.957 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:47:59.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:47:59.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:47:59.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:00.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:01.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:01.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:01.387 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:48:01.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:48:01.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:01.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:01.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:02.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:02.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:02.911 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:48:02.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:48:02.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:02.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:03.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:04.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:04.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:04.327 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:48:04.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:48:04.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:04.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:05.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:06.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:06.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:06.243 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:48:06.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:48:06.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:06.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:06.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:07.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:07.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:07.880 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:48:07.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:48:07.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:07.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:08.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:10.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:10.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:10.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:48:10.001 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:48:10.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:10.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:10.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:11.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:11.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:11.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:48:11.745 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:48:11.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:11.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:12.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:13.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:13.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:13.696 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:48:13.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:48:13.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:13.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:14.247 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2015313222,v1:127.0.0.1:6803/2015313222] [v2:127.0.0.1:6804/2015313222,v1:127.0.0.1:6805/2015313222] exists,up dcc693c4-02b2-42ea-881d-5a8f6679dd56 2024-09-16T15:48:14.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:48:14.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:48:14.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:48:14.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2024-09-16T15:48:14.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2024-09-16T15:48:14.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2024-09-16T15:48:14.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2024-09-16T15:48:14.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2024-09-16T15:48:14.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2024-09-16T15:48:14.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-09-16T15:48:14.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2024-09-16T15:48:14.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2024-09-16T15:48:14.424 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T15:48:14.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2024-09-16T15:48:14.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T15:48:14.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T15:48:14.439 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:48:14.439 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:14.439 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:48:14.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154844/ceph-osd.0.asok 2024-09-16T15:48:14.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2024-09-16T15:48:14.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.154844/ceph-osd.0.asok log flush 2024-09-16T15:48:14.546 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T15:48:14.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-09-16T15:48:14.563 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T15:48:14.412+0000 7f0f9575b640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2024-09-16T15:48:14.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2024-09-16T15:48:14.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2024-09-16T15:48:14.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T15:48:14.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T15:48:14.566 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:48:14.566 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:14.566 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:48:14.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154844/ceph-osd.0.asok 2024-09-16T15:48:14.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2024-09-16T15:48:14.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.154844/ceph-osd.0.asok log reopen 2024-09-16T15:48:14.692 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T15:48:14.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-09-16T15:48:14.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-16T15:48:14.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:48:14.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-16T15:48:14.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:48:14.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:48:14.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:48:14.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:48:14.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:48:15.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:48:15.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:48:15.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:48:15.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:48:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:48:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:48:15.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:48:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:48:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:48:15.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:48:15.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:48:15.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:48:15.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:48:15.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-16T15:48:15.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:48:15.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:15.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:48:15.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154844 2024-09-16T15:48:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:48:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:48:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-09-16T15:48:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-09-16T15:48:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-09-16T15:48:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-09-16T15:48:15.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-16T15:48:15.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:48:15.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-16T15:48:15.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:48:15.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:48:15.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:48:15.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:48:15.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:48:15.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:48:15.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:48:15.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:48:15.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:48:15.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:48:15.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:48:15.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:48:15.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:48:15.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:48:15.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:48:15.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:48:15.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:48:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:48:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-16T15:48:15.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:48:15.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:15.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:48:15.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154844 2024-09-16T15:48:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:48:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:48:15.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-09-16T15:48:15.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:48:15.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:15.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:48:15.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.154844 2024-09-16T15:48:15.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:48:15.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T15:48:15.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:48:15.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-09-16T15:48:15.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2024-09-16T15:48:15.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2024-09-16T15:48:15.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2024-09-16T15:48:15.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2024-09-16T15:48:15.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2024-09-16T15:48:15.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2024-09-16T15:48:15.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2024-09-16T15:48:15.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2024-09-16T15:48:15.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2024-09-16T15:48:15.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2024-09-16T15:48:15.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2024-09-16T15:48:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2024-09-16T15:48:15.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2024-09-16T15:48:15.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2024-09-16T15:48:15.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2024-09-16T15:48:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2024-09-16T15:48:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2024-09-16T15:48:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2024-09-16T15:48:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2024-09-16T15:48:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-09-16T15:48:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-16T15:48:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:48:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-16T15:48:15.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:48:15.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:48:15.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:48:15.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:48:15.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:48:15.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:48:15.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:48:15.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:48:15.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:48:15.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:48:15.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:48:15.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:48:15.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:48:15.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:48:15.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:48:15.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:48:15.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:48:15.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:48:15.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-16T15:48:15.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:48:15.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:15.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:48:15.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154844 2024-09-16T15:48:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:48:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:48:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T15:48:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-config 0 2024-09-16T15:48:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-16T15:48:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T15:48:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-16T15:48:15.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:48:15.206 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:48:15.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:48:15.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:48:15.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:48:15.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:48:15.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:48:15.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:48:15.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:48:15.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:48:15.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:48:15.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:48:15.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:48:15.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:48:15.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:48:15.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:48:15.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:48:15.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T15:48:15.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-16T15:48:15.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:48:15.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:15.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154844 2024-09-16T15:48:15.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154844 2024-09-16T15:48:15.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:48:15.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:48:15.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T15:48:15.219 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T15:48:15.219 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T15:48:15.271 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2024-09-16T15:48:15.271 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2024-09-16T15:48:15.324 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T15:48:15.329 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T15:48:15.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-copy-from 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2024-09-16T15:48:15.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2024-09-16T15:48:15.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2024-09-16T15:48:15.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=366f69d9-7a36-46a1-94db-d57dfc238d90 --auth-supported=none ' 2024-09-16T15:48:15.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2024-09-16T15:48:15.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2024-09-16T15:48:15.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T15:48:15.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2024-09-16T15:48:15.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2024-09-16T15:48:15.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2024-09-16T15:48:15.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2024-09-16T15:48:15.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2024-09-16T15:48:15.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-09-16T15:48:15.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:48:15.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-09-16T15:48:15.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:48:15.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:48:15.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:48:15.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:48:15.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:48:15.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:48:15.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:48:15.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:48:15.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:48:15.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:48:15.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:48:15.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:48:15.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:48:15.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:48:15.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:48:15.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:48:15.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:48:15.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:48:15.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-09-16T15:48:15.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:48:15.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:15.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158884 2024-09-16T15:48:15.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.158884 2024-09-16T15:48:15.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:48:15.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:48:15.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2024-09-16T15:48:15.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:48:15.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:15.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158884 2024-09-16T15:48:15.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.158884 2024-09-16T15:48:15.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:48:15.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T15:48:15.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T15:48:15.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:48:15.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2024-09-16T15:48:15.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2024-09-16T15:48:15.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2024-09-16T15:48:15.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2024-09-16T15:48:15.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2024-09-16T15:48:15.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:48:15.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:48:15.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:48:15.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2024-09-16T15:48:15.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2024-09-16T15:48:15.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:48:15.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:48:15.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:48:15.692 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:48:15.692 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:15.692 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158884 2024-09-16T15:48:15.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158884/$cluster-$name.asok' 2024-09-16T15:48:15.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.158884/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:48:16.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:48:16.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:48:16.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:48:16.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:48:16.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:48:16.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:48:16.061 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:48:16.061 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:48:16.061 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:48:16.061 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:48:16.061 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:16.061 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158884 2024-09-16T15:48:16.061 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.158884/ceph-mon.a.asok 2024-09-16T15:48:16.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:48:16.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.158884/ceph-mon.a.asok config get fsid 2024-09-16T15:48:16.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:48:16.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:48:16.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:48:16.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:48:16.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:48:16.182 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:48:16.182 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:48:16.182 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:48:16.183 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:48:16.183 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:16.183 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158884 2024-09-16T15:48:16.183 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.158884/ceph-mon.a.asok 2024-09-16T15:48:16.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:48:16.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.158884/ceph-mon.a.asok config get mon_host 2024-09-16T15:48:16.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2024-09-16T15:48:16.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2024-09-16T15:48:16.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:48:16.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:48:16.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:48:16.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2024-09-16T15:48:16.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:48:17.195 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:48:17.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:48:17.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:48:17.196 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:48:17.196 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:17.196 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158884 2024-09-16T15:48:17.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158884/$cluster-$name.asok' 2024-09-16T15:48:17.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:48:17.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.158884/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:48:17.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2024-09-16T15:48:17.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=366f69d9-7a36-46a1-94db-d57dfc238d90 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-09-16T15:48:17.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158884 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158884/$cluster-$name.asok' 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.158884/$cluster-$name.asok' 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:48:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:48:17.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:48:17.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:48:17.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:48:17.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/0 2024-09-16T15:48:17.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:48:17.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b60385f-0df0-4ee6-85fa-fe1a1c044138 2024-09-16T15:48:17.239 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 4b60385f-0df0-4ee6-85fa-fe1a1c044138 2024-09-16T15:48:17.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4b60385f-0df0-4ee6-85fa-fe1a1c044138' 2024-09-16T15:48:17.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:48:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBU+hmW6FODxAAFDOI2rWWAGqWElupecYoog== 2024-09-16T15:48:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBU+hmW6FODxAAFDOI2rWWAGqWElupecYoog=="}' 2024-09-16T15:48:17.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b60385f-0df0-4ee6-85fa-fe1a1c044138 -i td/osd-copy-from/0/new.json 2024-09-16T15:48:17.615 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:48:17.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/0/new.json 2024-09-16T15:48:17.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=366f69d9-7a36-46a1-94db-d57dfc238d90 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158884/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBU+hmW6FODxAAFDOI2rWWAGqWElupecYoog== --osd-uuid 4b60385f-0df0-4ee6-85fa-fe1a1c044138 2024-09-16T15:48:17.667 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:17.664+0000 7fd25dc037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:17.670 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:17.668+0000 7fd25dc037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:17.671 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:17.668+0000 7fd25dc037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:17.672 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:17.672+0000 7fd25dc037c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-09-16T15:48:17.672 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:17.672+0000 7fd25dc037c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-09-16T15:48:17.672 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:17.672+0000 7fd25dc037c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-09-16T15:48:17.777 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:17.776+0000 7fd25dc037c0 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2024-09-16T15:48:23.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/0/keyring 2024-09-16T15:48:23.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:48:23.621 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:48:23.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:48:23.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:48:24.185 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:48:24.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:48:24.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=366f69d9-7a36-46a1-94db-d57dfc238d90 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158884/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:48:24.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:48:24.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:48:24.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:48:24.216 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:24.216+0000 7f004cfb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:24.244 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:24.244+0000 7f004cfb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:24.246 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:24.244+0000 7f004cfb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:48:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:48:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:48:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:48:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:48:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:48:24.650 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:48:24.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:24.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:24.821 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:24.820+0000 7f004cfb17c0 -1 Falling back to public interface 2024-09-16T15:48:25.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:26.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:26.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:26.119 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:48:26.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:48:26.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:26.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:26.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:27.352 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:27.352+0000 7f004cfb17c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:48:27.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:27.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:27.550 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:48:27.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:48:27.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:27.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:28.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:29.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:29.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:29.058 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:48:29.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:48:29.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:29.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:29.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:30.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:30.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:30.494 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:48:30.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:48:30.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:30.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:31.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:32.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:32.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:32.040 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:48:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:48:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:32.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:32.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:33.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:33.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:33.491 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:48:33.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:48:33.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:33.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:34.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:35.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:35.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:35.038 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:48:35.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:48:35.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:35.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:35.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:36.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:36.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:36.471 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:48:36.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:48:36.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:36.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:36.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:37.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:37.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:37.905 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:48:37.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:48:37.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:37.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:38.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:39.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:39.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:39.381 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:48:39.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:48:39.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:39.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:39.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:40.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:40.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:40.894 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:48:40.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:48:40.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:40.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:41.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:42.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:42.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:42.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:48:42.641 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:48:42.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:42.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:43.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:44.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:44.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:44.572 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:48:44.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:48:44.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:44.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:45.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:46.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:46.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:46.548 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:48:46.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:48:46.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:46.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:47.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:48.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:48.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:48.836 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:48:48.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:48:48.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:48.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:48:49.340 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3873429457,v1:127.0.0.1:6803/3873429457] [v2:127.0.0.1:6804/3873429457,v1:127.0.0.1:6805/3873429457] exists,up 4b60385f-0df0-4ee6-85fa-fe1a1c044138 2024-09-16T15:48:49.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:48:49.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:48:49.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:48:49.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2024-09-16T15:48:49.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-09-16T15:48:49.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:48:49.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=366f69d9-7a36-46a1-94db-d57dfc238d90 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:48:49.343 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:48:49.344 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:48:49.344 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158884 2024-09-16T15:48:49.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158884/$cluster-$name.asok' 2024-09-16T15:48:49.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.158884/$cluster-$name.asok' 2024-09-16T15:48:49.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:48:49.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/1 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=88994e57-32e0-4895-b1fc-595d684669c5 2024-09-16T15:48:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 88994e57-32e0-4895-b1fc-595d684669c5' 2024-09-16T15:48:49.347 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 88994e57-32e0-4895-b1fc-595d684669c5 2024-09-16T15:48:49.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:48:49.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhU+hmBQTtFRAA/BEZGuTqpo9ElZOSJS+o7g== 2024-09-16T15:48:49.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhU+hmBQTtFRAA/BEZGuTqpo9ElZOSJS+o7g=="}' 2024-09-16T15:48:49.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 88994e57-32e0-4895-b1fc-595d684669c5 -i td/osd-copy-from/1/new.json 2024-09-16T15:48:49.893 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:48:49.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/1/new.json 2024-09-16T15:48:49.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=366f69d9-7a36-46a1-94db-d57dfc238d90 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158884/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhU+hmBQTtFRAA/BEZGuTqpo9ElZOSJS+o7g== --osd-uuid 88994e57-32e0-4895-b1fc-595d684669c5 2024-09-16T15:48:49.945 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:49.944+0000 7f7cd5f6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:49.947 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:49.944+0000 7f7cd5f6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:49.948 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:49.948+0000 7f7cd5f6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:49.948 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:49.948+0000 7f7cd5f6e7c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-09-16T15:48:49.949 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:49.948+0000 7f7cd5f6e7c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-09-16T15:48:49.949 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:49.948+0000 7f7cd5f6e7c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-09-16T15:48:50.073 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:50.076+0000 7f7cd5f6e7c0 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2024-09-16T15:48:55.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/1/keyring 2024-09-16T15:48:55.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:48:55.804 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:48:55.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:48:55.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:48:56.371 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:48:56.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:48:56.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=366f69d9-7a36-46a1-94db-d57dfc238d90 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158884/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:48:56.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:48:56.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:48:56.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:48:56.401 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:56.400+0000 7f9aef4337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:56.423 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:56.420+0000 7f9aef4337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:56.425 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:56.424+0000 7f9aef4337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:48:56.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:48:56.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:48:56.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:48:56.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:48:56.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:48:56.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:56.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:48:56.822 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:48:56.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:56.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:48:56.989 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:56.988+0000 7f9aef4337c0 -1 Falling back to public interface 2024-09-16T15:48:57.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:58.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:58.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:58.270 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:48:58.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:48:58.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:58.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:48:58.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:48:59.481 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:48:59.480+0000 7f9aef4337c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:48:59.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:48:59.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:48:59.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:48:59.743 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:48:59.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:48:59.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:00.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:01.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:01.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:01.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:49:01.272 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:49:01.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:01.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:01.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:02.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:02.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:02.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:49:02.698 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:49:02.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:02.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:03.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:04.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:04.231 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:49:04.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:04.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:49:04.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:04.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:04.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:05.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:05.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:05.771 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:49:05.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:49:05.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:05.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:06.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:07.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:07.345 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:49:07.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:07.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:49:07.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:07.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:07.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:08.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:08.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:08.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:49:08.783 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:49:08.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:08.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:09.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:10.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:10.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:10.284 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:49:10.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:49:10.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:10.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:10.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:11.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:11.750 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:49:11.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:11.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:49:11.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:11.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:12.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:13.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:13.411 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:49:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:49:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:14.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:15.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:15.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:15.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:49:15.190 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:49:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:15.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:16.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:16.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:16.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:49:16.974 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:49:16.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:16.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:17.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:18.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:18.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:18.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:49:18.726 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:49:18.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:18.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:19.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:20.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:20.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:20.793 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:49:20.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:49:20.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:20.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:21.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:22.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:22.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:22.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:49:22.543 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:49:22.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:22.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:49:23.053 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1726160074,v1:127.0.0.1:6811/1726160074] [v2:127.0.0.1:6812/1726160074,v1:127.0.0.1:6813/1726160074] exists,up 88994e57-32e0-4895-b1fc-595d684669c5 2024-09-16T15:49:23.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:49:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:49:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:49:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2024-09-16T15:49:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T15:49:23.585 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-09-16T15:49:23.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T15:49:23.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T15:49:24.155 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' already exists 2024-09-16T15:49:24.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:49:25.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T15:49:26.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2024-09-16T15:49:26.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2024-09-16T15:49:27.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2024-09-16T15:49:27.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2024-09-16T15:49:27.352 INFO:tasks.workunit.client.0.smithi039.stdout:rbd/foo2 mtime 2024-09-16T15:49:26.000000+0000, size 576136 2024-09-16T15:49:27.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2024-09-16T15:49:27.494 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T15:49:27.523 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T15:49:27.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2024-09-16T15:49:27.573 INFO:tasks.workunit.client.0.smithi039.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2024-09-16T15:49:27.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2024-09-16T15:49:27.620 INFO:tasks.workunit.client.0.smithi039.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2024-09-16T15:49:27.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2024-09-16T15:49:27.757 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T15:49:27.785 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T15:49:27.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2024-09-16T15:49:27.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2024-09-16T15:49:28.020 INFO:tasks.workunit.client.0.smithi039.stdout:rbd/foo3 mtime 2024-09-16T15:49:26.000000+0000, size 576136 2024-09-16T15:49:28.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2024-09-16T15:49:28.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-09-16T15:49:28.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:49:28.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-09-16T15:49:28.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:49:28.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:49:28.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:49:28.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:49:28.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:49:28.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:49:28.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:49:28.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:49:28.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:49:28.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:49:28.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:49:28.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:49:28.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:49:28.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:49:28.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:49:28.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:49:28.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:49:28.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:49:28.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-09-16T15:49:28.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:49:28.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:28.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158884 2024-09-16T15:49:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.158884 2024-09-16T15:49:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:49:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:49:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T15:49:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-copy-from 0 2024-09-16T15:49:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-09-16T15:49:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T15:49:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-09-16T15:49:28.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:49:28.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:49:28.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:49:28.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:49:28.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:49:28.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:49:28.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:49:28.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:49:28.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:49:28.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:49:28.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:49:28.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:49:28.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:49:28.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:49:28.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:49:28.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:49:28.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:49:28.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T15:49:28.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-09-16T15:49:28.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:49:28.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:28.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158884 2024-09-16T15:49:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.158884 2024-09-16T15:49:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:49:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:49:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T15:49:28.163 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T15:49:28.163 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T15:49:28.218 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2024-09-16T15:49:28.218 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2024-09-16T15:49:28.270 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T15:49:28.276 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T15:49:28.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:49:28.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:49:28.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-dup 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2024-09-16T15:49:28.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2024-09-16T15:49:28.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=19e436c0-49d4-421e-880f-a314b5031444 --auth-supported=none ' 2024-09-16T15:49:28.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-16T15:49:28.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2024-09-16T15:49:28.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-16T15:49:28.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2024-09-16T15:49:28.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T15:49:28.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs= 2024-09-16T15:49:28.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T15:49:28.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-dup 0 2024-09-16T15:49:28.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2024-09-16T15:49:28.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T15:49:28.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2024-09-16T15:49:28.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:49:28.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:49:28.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:49:28.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:49:28.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:49:28.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:49:28.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:49:28.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:49:28.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:49:28.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:49:28.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:49:28.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:49:28.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:49:28.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:49:28.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:49:28.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:49:28.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:49:28.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T15:49:28.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2024-09-16T15:49:28.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:49:28.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:28.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162842 2024-09-16T15:49:28.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162842 2024-09-16T15:49:28.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:49:28.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:49:28.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T15:49:28.292 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T15:49:28.292 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T15:49:28.342 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2024-09-16T15:49:28.342 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2024-09-16T15:49:28.397 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T15:49:28.402 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T15:49:28.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:49:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:49:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T15:49:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T15:49:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T15:49:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T15:49:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T15:49:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T15:49:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-fast-mark-down 2024-09-16T15:49:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2024-09-16T15:49:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2024-09-16T15:49:28.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2024-09-16T15:49:28.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2024-09-16T15:49:28.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2024-09-16T15:49:28.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2024-09-16T15:49:28.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2024-09-16T15:49:28.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none ' 2024-09-16T15:49:28.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none ' 2024-09-16T15:49:28.405 INFO:tasks.workunit.client.0.smithi039.stdout:Ensuring old behavior is there... 2024-09-16T15:49:28.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2024-09-16T15:49:28.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:49:28.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:49:28.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:49:28.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:49:28.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:49:28.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:49:28.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:49:28.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:49:28.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:49:28.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:49:28.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:49:28.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:49:28.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:49:28.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:49:28.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:49:28.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-16T15:49:28.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:49:28.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:28.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:49:28.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162871 2024-09-16T15:49:28.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:49:28.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:49:28.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-09-16T15:49:28.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:49:28.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:28.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:49:28.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.162871 2024-09-16T15:49:28.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:49:28.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T15:49:28.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T15:49:28.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:49:28.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-09-16T15:49:28.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-09-16T15:49:28.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-09-16T15:49:28.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:49:28.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:49:28.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:49:28.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-09-16T15:49:28.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-09-16T15:49:28.450 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:28.447+0000 7f740609cc40 -1 failed for service _ceph-mon._tcp 2024-09-16T15:49:28.450 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:28.447+0000 7f740609cc40 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-16T15:49:28.450 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T15:49:28.450 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:28.447+0000 7f740609cc40 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2024-09-16T15:49:28.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:49:28.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:49:28.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:49:28.826 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:49:28.826 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:28.826 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:49:28.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162871/$cluster-$name.asok' 2024-09-16T15:49:28.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-16T15:49:29.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:29.011+0000 7fdb8be1bc40 -1 failed for service _ceph-mon._tcp 2024-09-16T15:49:29.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:29.011+0000 7fdb8be1bc40 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-16T15:49:29.015 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T15:49:29.016 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:29.011+0000 7fdb8be1bc40 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2024-09-16T15:49:29.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2024-09-16T15:49:29.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2024-09-16T15:49:29.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-09-16T15:49:29.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-09-16T15:49:29.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-16T15:49:29.024 INFO:tasks.workunit.client.0.smithi039.stdout:Testing async msgr... 2024-09-16T15:49:29.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2024-09-16T15:49:29.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2024-09-16T15:49:29.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-09-16T15:49:29.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-09-16T15:49:29.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-09-16T15:49:29.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-16T15:49:29.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:49:29.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-16T15:49:29.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:49:29.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:49:29.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:49:29.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:49:29.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:49:29.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:49:29.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:49:29.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:49:29.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:49:29.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:49:29.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:49:29.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:49:29.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:49:29.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:49:29.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:49:29.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:49:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:49:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:49:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-16T15:49:29.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:49:29.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:29.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:49:29.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162871 2024-09-16T15:49:29.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:49:29.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:49:29.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-09-16T15:49:29.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:49:29.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:29.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:49:29.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.162871 2024-09-16T15:49:29.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:49:29.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T15:49:29.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:49:29.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-09-16T15:49:29.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-09-16T15:49:29.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-09-16T15:49:29.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:49:29.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:49:29.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:49:29.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-09-16T15:49:29.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-09-16T15:49:29.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:49:29.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:49:29.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:49:29.413 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:49:29.413 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:29.414 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:49:29.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162871/$cluster-$name.asok' 2024-09-16T15:49:29.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-16T15:49:29.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:49:29.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:49:29.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:49:29.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:49:29.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:49:29.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:49:29.636 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:49:29.636 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:49:29.636 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:49:29.637 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:49:29.637 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:29.637 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:49:29.637 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162871/ceph-mon.a.asok 2024-09-16T15:49:29.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:49:29.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162871/ceph-mon.a.asok config get fsid 2024-09-16T15:49:29.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:49:29.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:49:29.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:49:29.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:49:29.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:49:29.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:49:29.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:49:29.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:49:29.757 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:49:29.757 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:29.757 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:49:29.757 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162871/ceph-mon.a.asok 2024-09-16T15:49:29.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:49:29.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162871/ceph-mon.a.asok config get mon_host 2024-09-16T15:49:29.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2024-09-16T15:49:29.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2024-09-16T15:49:29.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:49:29.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:49:29.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:49:29.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2024-09-16T15:49:29.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:49:30.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:49:30.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:49:30.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:49:30.767 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:49:30.767 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:30.767 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:49:30.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162871/$cluster-$name.asok' 2024-09-16T15:49:30.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:49:30.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:49:30.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-09-16T15:49:30.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2024-09-16T15:49:30.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-09-16T15:49:30.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-09-16T15:49:30.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:49:30.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:49:30.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:49:30.833 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:49:30.833 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:49:30.833 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:49:30.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162871/$cluster-$name.asok' 2024-09-16T15:49:30.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' 2024-09-16T15:49:30.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:49:30.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:49:30.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:49:30.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-09-16T15:49:30.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-09-16T15:49:30.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:49:30.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:49:30.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:49:30.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:49:30.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:49:30.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:49:30.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/0 2024-09-16T15:49:30.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:49:30.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=597b8109-0bb9-4c7f-a42f-f15fb1d1152b 2024-09-16T15:49:30.835 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 597b8109-0bb9-4c7f-a42f-f15fb1d1152b 2024-09-16T15:49:30.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 597b8109-0bb9-4c7f-a42f-f15fb1d1152b' 2024-09-16T15:49:30.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:49:30.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCKU+hmPaj2MhAAWbh5Rf4wvOP75vg1n4IXjQ== 2024-09-16T15:49:30.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCKU+hmPaj2MhAAWbh5Rf4wvOP75vg1n4IXjQ=="}' 2024-09-16T15:49:30.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 597b8109-0bb9-4c7f-a42f-f15fb1d1152b -i td/osd-fast-mark-down/0/new.json 2024-09-16T15:49:31.181 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:49:31.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/0/new.json 2024-09-16T15:49:31.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKU+hmPaj2MhAAWbh5Rf4wvOP75vg1n4IXjQ== --osd-uuid 597b8109-0bb9-4c7f-a42f-f15fb1d1152b 2024-09-16T15:49:31.231 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:31.231+0000 7f9f19bd27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:49:31.234 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:31.235+0000 7f9f19bd27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:49:31.235 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:31.235+0000 7f9f19bd27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:49:31.235 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:31.235+0000 7f9f19bd27c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-09-16T15:49:31.235 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:31.235+0000 7f9f19bd27c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-09-16T15:49:31.235 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:31.235+0000 7f9f19bd27c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-09-16T15:49:31.312 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:31.315+0000 7f9f19bd27c0 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2024-09-16T15:49:37.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2024-09-16T15:49:37.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:49:37.135 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:49:37.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:49:37.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:49:37.707 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:49:37.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:49:37.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:49:37.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:49:37.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:49:37.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:49:37.741 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:37.743+0000 7f9d40b737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:49:37.743 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:37.743+0000 7f9d40b737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:49:37.744 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:37.743+0000 7f9d40b737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:49:38.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:49:38.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:49:38.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:49:38.175 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:49:38.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:49:38.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:49:38.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:38.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:49:38.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:38.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:38.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:38.303+0000 7f9d40b737c0 -1 Falling back to public interface 2024-09-16T15:49:38.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:39.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:39.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:39.612 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:49:39.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:49:39.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:39.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:40.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:40.755 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:49:40.754+0000 7f9d40b737c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:49:41.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:41.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:41.048 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:49:41.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:49:41.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:41.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:41.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:42.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:42.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:42.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:49:42.677 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:49:42.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:42.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:43.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:44.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:44.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:44.113 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:49:44.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:49:44.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:44.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:44.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:45.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:45.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:45.541 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:49:45.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:49:45.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:45.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:46.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:47.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:47.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:47.004 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:49:47.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:49:47.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:47.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:47.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:48.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:48.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:48.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:49:48.565 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:49:48.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:48.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:49.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:50.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:50.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:50.038 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:49:50.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:49:50.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:50.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:50.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:51.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:51.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:51.566 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:49:51.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:49:51.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:51.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:52.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:53.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:53.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:53.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:49:53.031 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:49:53.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:53.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:54.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:55.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:55.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:49:55.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:49:55.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:55.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:55.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:56.550 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:49:56.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:49:56.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:56.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:57.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:49:58.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:49:58.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:49:58.222 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:49:58.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:49:58.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:49:58.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:49:59.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:00.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:00.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:00.094 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:50:00.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:50:00.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:00.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:50:01.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:02.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:02.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:02.226 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:50:02.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:50:02.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:02.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:50:02.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:03.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:03.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:03.986 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:50:03.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:50:03.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:03.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:50:04.462 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2448723298,v1:127.0.0.1:6803/2448723298] [v2:127.0.0.1:6804/2448723298,v1:127.0.0.1:6805/2448723298] exists,up 597b8109-0bb9-4c7f-a42f-f15fb1d1152b 2024-09-16T15:50:04.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:50:04.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:50:04.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:50:04.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2024-09-16T15:50:04.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=164017 2024-09-16T15:50:04.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-09-16T15:50:04.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2024-09-16T15:50:04.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-09-16T15:50:04.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:50:04.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162871/$cluster-$name.asok' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:50:04.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:50:04.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/1 2024-09-16T15:50:04.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:50:04.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4d811059-27aa-4f91-b2c4-29de841c28b1 2024-09-16T15:50:04.469 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 4d811059-27aa-4f91-b2c4-29de841c28b1 2024-09-16T15:50:04.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4d811059-27aa-4f91-b2c4-29de841c28b1' 2024-09-16T15:50:04.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:50:04.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCsU+hmtY6CHRAAIURc7ee/ps8gOtWXzM5koA== 2024-09-16T15:50:04.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCsU+hmtY6CHRAAIURc7ee/ps8gOtWXzM5koA=="}' 2024-09-16T15:50:04.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4d811059-27aa-4f91-b2c4-29de841c28b1 -i td/osd-fast-mark-down/1/new.json 2024-09-16T15:50:05.033 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:50:05.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/1/new.json 2024-09-16T15:50:05.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCsU+hmtY6CHRAAIURc7ee/ps8gOtWXzM5koA== --osd-uuid 4d811059-27aa-4f91-b2c4-29de841c28b1 2024-09-16T15:50:05.087 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:05.086+0000 7fd256cc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:05.089 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:05.090+0000 7fd256cc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:05.090 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:05.090+0000 7fd256cc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:05.090 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:05.090+0000 7fd256cc37c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-09-16T15:50:05.090 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:05.090+0000 7fd256cc37c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-09-16T15:50:05.091 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:05.090+0000 7fd256cc37c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-09-16T15:50:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:05.122+0000 7fd256cc37c0 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2024-09-16T15:50:11.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2024-09-16T15:50:11.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:50:11.156 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:50:11.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:50:11.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:50:11.727 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:50:11.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:50:11.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:50:11.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:50:11.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:50:11.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:50:11.762 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:11.762+0000 7f50219487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:11.765 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:11.766+0000 7f50219487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:11.766 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:11.766+0000 7f50219487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:12.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:50:12.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:50:12.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:50:12.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:50:12.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:50:12.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:12.172 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:50:12.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:50:12.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:12.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:12.342 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:12.342+0000 7f50219487c0 -1 Falling back to public interface 2024-09-16T15:50:12.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:13.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:13.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:13.630 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:50:13.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:50:13.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:13.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:14.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:14.859 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:14.858+0000 7f50219487c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:50:15.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:15.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:15.099 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:50:15.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:50:15.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:15.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:15.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:16.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:16.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:16.630 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:50:16.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:50:16.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:16.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:17.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:18.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:18.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:18.169 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:50:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:50:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:18.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:19.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:19.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:19.654 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:50:19.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:50:19.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:19.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:20.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:21.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:21.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:21.164 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:50:21.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:50:21.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:21.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:21.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:22.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:22.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:22.734 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:50:22.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:50:22.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:22.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:23.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:24.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:24.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:24.185 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:50:24.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:50:24.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:24.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:24.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:25.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:25.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:25.738 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:50:25.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:50:25.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:25.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:26.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:27.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:27.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:27.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:50:27.306 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:50:27.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:27.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:28.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:29.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:29.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:29.285 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:50:29.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:50:29.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:29.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:29.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:30.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:30.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:30.938 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:50:30.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:50:30.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:30.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:31.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:32.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:32.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:32.632 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:50:32.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:50:32.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:32.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:33.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:34.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:34.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:34.322 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:50:34.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:50:34.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:34.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:35.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:36.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:36.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:36.393 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:50:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:50:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:37.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:38.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:38.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:38.383 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:50:38.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:50:38.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:38.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:50:38.852 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2772634673,v1:127.0.0.1:6811/2772634673] [v2:127.0.0.1:6812/2772634673,v1:127.0.0.1:6813/2772634673] exists,up 4d811059-27aa-4f91-b2c4-29de841c28b1 2024-09-16T15:50:38.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:50:38.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:50:38.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:50:38.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2024-09-16T15:50:38.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=165733 2024-09-16T15:50:38.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-09-16T15:50:38.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2024-09-16T15:50:38.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-09-16T15:50:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:50:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T15:50:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:50:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2024-09-16T15:50:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-16T15:50:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:50:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:50:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:50:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2024-09-16T15:50:38.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2024-09-16T15:50:38.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:50:38.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:50:38.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-09-16T15:50:38.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:50:38.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:50:38.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:50:38.860 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:50:38.860 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:50:38.860 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:50:38.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162871/$cluster-$name.asok' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:50:38.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:50:38.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/2 2024-09-16T15:50:38.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:50:38.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=533aa2f9-0b65-4f45-80f7-48bb42f7e788 2024-09-16T15:50:38.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 533aa2f9-0b65-4f45-80f7-48bb42f7e788' 2024-09-16T15:50:38.862 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 533aa2f9-0b65-4f45-80f7-48bb42f7e788 2024-09-16T15:50:38.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:50:38.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOU+hmJIDxNBAAir2y17R6IwVeiBM4ru+SjA== 2024-09-16T15:50:38.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOU+hmJIDxNBAAir2y17R6IwVeiBM4ru+SjA=="}' 2024-09-16T15:50:38.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 533aa2f9-0b65-4f45-80f7-48bb42f7e788 -i td/osd-fast-mark-down/2/new.json 2024-09-16T15:50:39.446 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:50:39.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/2/new.json 2024-09-16T15:50:39.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOU+hmJIDxNBAAir2y17R6IwVeiBM4ru+SjA== --osd-uuid 533aa2f9-0b65-4f45-80f7-48bb42f7e788 2024-09-16T15:50:39.499 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:39.497+0000 7f958a4a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:39.526 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:39.525+0000 7f958a4a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:39.528 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:39.525+0000 7f958a4a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:39.528 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:39.525+0000 7f958a4a67c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-09-16T15:50:39.528 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:39.525+0000 7f958a4a67c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-09-16T15:50:39.529 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:39.525+0000 7f958a4a67c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-09-16T15:50:39.618 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:39.617+0000 7f958a4a67c0 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2024-09-16T15:50:45.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2024-09-16T15:50:45.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:50:45.447 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T15:50:45.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T15:50:45.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:50:46.002 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:50:46.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T15:50:46.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f8c6cfee-57ac-4777-8c16-552c5cfda30c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:50:46.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:50:46.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:50:46.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:50:46.037 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:46.037+0000 7f623e4c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:46.048 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:46.049+0000 7f623e4c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:46.050 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:46.049+0000 7f623e4c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:50:46.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T15:50:46.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:50:46.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:50:46.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:50:46.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:50:46.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:46.462 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:50:46.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:50:46.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:46.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:50:46.622 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:46.621+0000 7f623e4c17c0 -1 Falling back to public interface 2024-09-16T15:50:46.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:47.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:47.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:47.898 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:50:47.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:50:47.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:47.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:50:48.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:49.091 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:50:49.089+0000 7f623e4c17c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:50:49.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:49.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:49.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:50:49.367 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:50:49.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:49.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:50:49.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:50.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:50.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:50.870 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:50:50.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:50:50.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:50.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:50:51.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:52.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:52.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:52.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:50:52.311 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:50:52.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:52.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:50:52.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:53.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:53.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:53.846 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:50:53.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:50:53.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:53.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:50:54.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:55.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:55.388 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:50:55.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:50:55.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:55.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:50:56.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:57.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:50:57.060 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:50:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:50:57.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:58.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:58.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:58.486 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:50:58.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:50:58.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:58.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:50:58.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:50:59.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:50:59.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:50:59.985 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:50:59.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:50:59.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:50:59.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:51:00.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:01.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:01.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:01.443 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:51:01.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:51:01.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:01.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:51:02.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:03.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:03.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:03.187 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:51:03.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:51:03.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:03.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:51:03.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:04.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:04.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:04.977 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:51:04.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:51:04.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:04.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:51:05.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:06.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:06.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:06.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:51:06.625 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:51:06.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:06.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:51:07.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:08.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:08.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:08.515 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:51:08.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:51:08.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:08.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:51:09.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:10.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:10.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:10.244 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:51:10.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:51:10.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:10.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:51:10.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:11.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:11.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:11.964 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:51:11.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:51:11.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:11.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:51:12.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:13.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:13.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:13.510 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T15:51:13.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T15:51:13.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:13.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:51:13.973 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/157967057,v1:127.0.0.1:6819/157967057] [v2:127.0.0.1:6820/157967057,v1:127.0.0.1:6821/157967057] exists,up 533aa2f9-0b65-4f45-80f7-48bb42f7e788 2024-09-16T15:51:13.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:51:13.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:51:13.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:51:13.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2024-09-16T15:51:13.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=167447 2024-09-16T15:51:13.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2024-09-16T15:51:13.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T15:51:14.449 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-09-16T15:51:14.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T15:51:14.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T15:51:15.040 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' already exists 2024-09-16T15:51:15.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:51:16.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T15:51:17.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2024-09-16T15:51:17.747 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-16T15:51:17.747 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2024-09-16T15:51:17.747 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_168417 2024-09-16T15:51:17.747 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T15:51:17.747 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-16T15:51:18.747 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 81 65 0.253875 0.253906 0.216859 0.203244 2024-09-16T15:51:19.747 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 1.70828 2024-09-16T15:51:19.747 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 128 2024-09-16T15:51:19.747 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4096 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4096 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.292693 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.253906 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.253906 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 74 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 0 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 65 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 65 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.212594 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.0324893 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 0.25118 2024-09-16T15:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.125338 2024-09-16T15:51:19.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2024-09-16T15:51:20.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2024-09-16T15:51:20.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2024-09-16T15:51:20.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-09-16T15:51:20.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2024-09-16T15:51:20.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=165733 2024-09-16T15:51:20.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 165733 -eq 0 ']' 2024-09-16T15:51:20.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=165733 2024-09-16T15:51:20.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 165733 2024-09-16T15:51:20.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-09-16T15:51:20.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-09-16T15:51:20.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-09-16T15:51:20.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-09-16T15:51:21.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-09-16T15:51:21.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-09-16T15:51:21.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-09-16T15:51:21.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-09-16T15:51:21.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 165733 -eq 165733 ']' 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=164017 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 164017 -eq 165733 ']' 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=164017 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 164017 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-09-16T15:51:22.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-09-16T15:51:22.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-09-16T15:51:22.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-09-16T15:51:23.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-09-16T15:51:23.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-09-16T15:51:23.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-09-16T15:51:23.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-09-16T15:51:23.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-09-16T15:51:23.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2024-09-16T15:51:23.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2024-09-16T15:51:23.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2024-09-16T15:51:23.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-09-16T15:51:23.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2024-09-16T15:51:23.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2024-09-16T15:51:23.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2024-09-16T15:51:23.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-16T15:51:23.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:51:23.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-16T15:51:23.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:51:23.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:51:23.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:51:23.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:51:23.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:51:24.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:51:24.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:51:24.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:51:24.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:51:24.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:51:24.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:51:24.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:51:24.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:51:24.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:51:24.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:51:24.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:51:24.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:51:24.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:51:24.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-16T15:51:24.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:51:24.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:51:24.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:51:24.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162871 2024-09-16T15:51:24.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:51:24.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:51:24.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2024-09-16T15:51:24.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T15:51:24.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-fast-mark-down 0 2024-09-16T15:51:24.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-16T15:51:24.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T15:51:24.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-16T15:51:24.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:51:24.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:51:24.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:51:24.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:51:24.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:51:24.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:51:24.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:51:24.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:51:24.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:51:24.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:51:24.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:51:24.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:51:24.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:51:24.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:51:24.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:51:24.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:51:24.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:51:24.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T15:51:24.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-16T15:51:24.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:51:24.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:51:24.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162871 2024-09-16T15:51:24.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162871 2024-09-16T15:51:24.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:51:24.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:51:24.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T15:51:24.124 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T15:51:24.124 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T15:51:24.178 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2024-09-16T15:51:24.179 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2024-09-16T15:51:24.235 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T15:51:24.240 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T15:51:24.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:51:24.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:51:24.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T15:51:24.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T15:51:24.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T15:51:24.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T15:51:24.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T15:51:24.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T15:51:24.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-force-create-pg 2024-09-16T15:51:24.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2024-09-16T15:51:24.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2024-09-16T15:51:24.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2024-09-16T15:51:24.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2024-09-16T15:51:24.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2024-09-16T15:51:24.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2024-09-16T15:51:24.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none ' 2024-09-16T15:51:24.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2024-09-16T15:51:24.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2024-09-16T15:51:24.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T15:51:24.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2024-09-16T15:51:24.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2024-09-16T15:51:24.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: setup td/osd-force-create-pg 2024-09-16T15:51:24.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2024-09-16T15:51:24.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2024-09-16T15:51:24.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-09-16T15:51:24.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:51:24.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-09-16T15:51:24.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:51:24.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:51:24.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:51:24.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:51:24.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:51:24.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:51:24.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:51:24.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:51:24.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:51:24.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:51:24.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:51:24.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:51:24.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:51:24.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:51:24.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:51:24.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:51:24.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:51:24.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:51:24.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-09-16T15:51:24.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:51:24.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:51:24.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:51:24.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168548 2024-09-16T15:51:24.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:51:24.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:51:24.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2024-09-16T15:51:24.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:51:24.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:51:24.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:51:24.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.168548 2024-09-16T15:51:24.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:51:24.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T15:51:24.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T15:51:24.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:51:24.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2024-09-16T15:51:24.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:16: run: TEST_reuse_id td/osd-force-create-pg 2024-09-16T15:51:24.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: local dir=td/osd-force-create-pg 2024-09-16T15:51:24.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T15:51:24.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2024-09-16T15:51:24.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:51:24.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:51:24.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:51:24.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2024-09-16T15:51:24.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T15:51:24.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:51:24.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:51:24.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:51:24.642 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:51:24.643 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:51:24.643 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:51:24.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:51:24.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T15:51:24.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:51:24.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:51:24.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:51:24.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:51:24.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:51:24.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:51:24.861 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:51:24.861 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:51:24.861 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:51:24.861 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:51:24.861 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:51:24.861 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:51:24.862 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.168548/ceph-mon.a.asok 2024-09-16T15:51:24.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:51:24.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.168548/ceph-mon.a.asok config get fsid 2024-09-16T15:51:24.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:51:24.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:51:24.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:51:24.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:51:24.981 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:51:24.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:51:24.981 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:51:24.981 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:51:24.981 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:51:24.981 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:51:24.981 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:51:24.982 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.168548/ceph-mon.a.asok 2024-09-16T15:51:24.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:51:24.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.168548/ceph-mon.a.asok config get mon_host 2024-09-16T15:51:25.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2024-09-16T15:51:25.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2024-09-16T15:51:25.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:51:25.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:51:25.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:51:25.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2024-09-16T15:51:25.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:51:26.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:51:26.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:51:26.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:51:26.006 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:51:26.006 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:51:26.006 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:51:26.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:51:26.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:51:26.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:51:26.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:51:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-16T15:51:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:51:26.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:51:26.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:51:26.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:51:26.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:51:26.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:51:26.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/0 2024-09-16T15:51:26.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:51:26.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c1d5ce29-cf35-4185-b2ff-cfb3f6b74701 2024-09-16T15:51:26.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c1d5ce29-cf35-4185-b2ff-cfb3f6b74701' 2024-09-16T15:51:26.042 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 c1d5ce29-cf35-4185-b2ff-cfb3f6b74701 2024-09-16T15:51:26.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:51:26.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+U+hm5C5XAxAAzOMlbFQ593oz4ngvgJzFVw== 2024-09-16T15:51:26.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+U+hm5C5XAxAAzOMlbFQ593oz4ngvgJzFVw=="}' 2024-09-16T15:51:26.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c1d5ce29-cf35-4185-b2ff-cfb3f6b74701 -i td/osd-force-create-pg/0/new.json 2024-09-16T15:51:26.408 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:51:26.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/0/new.json 2024-09-16T15:51:26.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+U+hm5C5XAxAAzOMlbFQ593oz4ngvgJzFVw== --osd-uuid c1d5ce29-cf35-4185-b2ff-cfb3f6b74701 2024-09-16T15:51:26.453 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:26.452+0000 7f894a14a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:51:26.455 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:26.452+0000 7f894a14a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:51:26.457 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:26.456+0000 7f894a14a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:51:26.457 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:26.456+0000 7f894a14a7c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-09-16T15:51:26.457 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:26.456+0000 7f894a14a7c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-09-16T15:51:26.457 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:26.456+0000 7f894a14a7c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-09-16T15:51:26.523 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:26.520+0000 7f894a14a7c0 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2024-09-16T15:51:32.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2024-09-16T15:51:32.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:51:32.318 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:51:32.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:51:32.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:51:32.878 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:51:32.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:51:32.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:51:32.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:51:32.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:51:32.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:51:32.911 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:32.908+0000 7f81857d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:51:32.920 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:32.916+0000 7f81857d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:51:32.921 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:32.920+0000 7f81857d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:51:33.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:51:33.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:51:33.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:51:33.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:51:33.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:51:33.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:33.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:51:33.306 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:51:33.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:33.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:33.495 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:33.492+0000 7f81857d97c0 -1 Falling back to public interface 2024-09-16T15:51:33.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:34.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:34.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:34.720 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:51:34.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:51:34.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:34.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:35.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:36.004 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:36.000+0000 7f81857d97c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:51:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:36.141 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:51:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:51:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:36.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:36.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:37.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:37.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:37.603 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:51:37.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:51:37.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:37.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:38.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:39.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:39.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:39.039 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:51:39.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:51:39.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:39.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:40.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:40.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:40.472 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:51:40.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:51:40.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:40.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:40.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:41.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:41.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:41.994 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:51:41.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:51:41.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:41.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:42.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:43.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:43.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:43.442 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:51:43.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:51:43.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:43.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:44.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:45.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:45.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:45.004 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:51:45.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:51:45.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:45.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:45.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:46.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:46.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:46.438 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:51:46.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:51:46.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:46.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:47.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:48.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:48.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:48.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:51:48.046 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:51:48.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:48.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:48.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:49.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:49.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:49.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:51:49.686 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:51:49.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:49.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:50.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:51.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:51.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:51.633 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:51:51.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:51:51.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:51.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:52.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:53.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:53.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:53.502 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:51:53.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:51:53.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:53.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:54.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:55.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:55.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:55.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:51:55.486 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:51:55.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:55.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:56.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:51:57.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:51:57.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:51:57.454 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:51:57.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:51:57.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:51:57.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:51:58.002 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1018592725,v1:127.0.0.1:6803/1018592725] [v2:127.0.0.1:6804/1018592725,v1:127.0.0.1:6805/1018592725] exists,up c1d5ce29-cf35-4185-b2ff-cfb3f6b74701 2024-09-16T15:51:58.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:51:58.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:51:58.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:51:58.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-16T15:51:58.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-16T15:51:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:51:58.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:51:58.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:51:58.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:51:58.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:51:58.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:51:58.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/1 2024-09-16T15:51:58.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:51:58.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c235452-bddc-4c87-9b02-9f2d120bab43 2024-09-16T15:51:58.009 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 7c235452-bddc-4c87-9b02-9f2d120bab43 2024-09-16T15:51:58.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7c235452-bddc-4c87-9b02-9f2d120bab43' 2024-09-16T15:51:58.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:51:58.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAeVOhmPumqARAAb8ci36BNrBoH32adt6FyxQ== 2024-09-16T15:51:58.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAeVOhmPumqARAAb8ci36BNrBoH32adt6FyxQ=="}' 2024-09-16T15:51:58.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c235452-bddc-4c87-9b02-9f2d120bab43 -i td/osd-force-create-pg/1/new.json 2024-09-16T15:51:58.581 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:51:58.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/1/new.json 2024-09-16T15:51:58.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAeVOhmPumqARAAb8ci36BNrBoH32adt6FyxQ== --osd-uuid 7c235452-bddc-4c87-9b02-9f2d120bab43 2024-09-16T15:51:58.635 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:58.631+0000 7fdd85e0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:51:58.638 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:58.635+0000 7fdd85e0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:51:58.639 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:58.635+0000 7fdd85e0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:51:58.640 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:58.635+0000 7fdd85e0b7c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-09-16T15:51:58.640 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:58.635+0000 7fdd85e0b7c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-09-16T15:51:58.640 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:58.635+0000 7fdd85e0b7c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-09-16T15:51:58.711 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:51:58.711+0000 7fdd85e0b7c0 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2024-09-16T15:52:04.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2024-09-16T15:52:04.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:52:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:52:04.763 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:52:04.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:52:05.369 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:52:05.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:52:05.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:52:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:52:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:52:05.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:52:05.400 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:05.395+0000 7fedd7e027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:52:05.403 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:05.399+0000 7fedd7e027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:52:05.404 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:05.399+0000 7fedd7e027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:52:05.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:52:05.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:52:05.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:52:05.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:52:05.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:52:05.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:05.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:52:05.867 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:52:05.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:05.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:05.998 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:05.995+0000 7fedd7e027c0 -1 Falling back to public interface 2024-09-16T15:52:06.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:07.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:07.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:07.349 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:52:07.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:52:07.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:07.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:07.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:08.534 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:08.531+0000 7fedd7e027c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:52:08.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:08.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:08.806 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:52:08.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:52:08.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:08.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:09.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:10.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:10.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:10.340 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:52:10.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:52:10.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:10.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:10.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:11.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:11.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:11.788 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:52:11.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:52:11.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:11.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:12.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:13.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:13.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:13.341 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:52:13.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:52:13.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:13.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:13.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:14.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:14.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:14.867 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:52:14.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:52:14.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:14.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:15.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:16.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:16.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:16.409 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:52:16.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:52:16.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:16.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:16.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:17.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:17.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:17.853 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:52:17.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:52:17.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:17.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:18.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:19.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:19.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:19.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:52:19.296 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:52:19.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:19.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:19.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:20.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:20.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:20.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:52:20.739 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:52:20.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:20.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:21.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:22.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:22.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:22.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:52:22.510 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:52:22.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:22.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:23.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:24.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:24.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:24.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:52:24.193 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:52:24.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:24.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:24.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:25.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:25.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:25.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:52:25.964 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:52:25.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:25.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:26.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:27.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:27.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:27.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:52:27.907 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:52:27.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:27.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:28.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:29.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:29.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:29.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:52:29.939 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:52:29.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:29.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:30.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:31.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:31.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:31.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:52:31.654 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:52:31.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:31.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:52:32.075 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/896352876,v1:127.0.0.1:6811/896352876] [v2:127.0.0.1:6812/896352876,v1:127.0.0.1:6813/896352876] exists,up 7c235452-bddc-4c87-9b02-9f2d120bab43 2024-09-16T15:52:32.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:28: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-09-16T15:52:32.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:52:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-16T15:52:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-16T15:52:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:52:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:52:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:52:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:52:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:52:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:52:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/2 2024-09-16T15:52:32.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:52:32.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c43d1489-abce-4ac6-9678-ee6d2cca26cc 2024-09-16T15:52:32.082 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 c43d1489-abce-4ac6-9678-ee6d2cca26cc 2024-09-16T15:52:32.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c43d1489-abce-4ac6-9678-ee6d2cca26cc' 2024-09-16T15:52:32.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:52:32.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAVOhmTo4zBhAA2K/OsCx3rnzu+g1PNNurRw== 2024-09-16T15:52:32.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAVOhmTo4zBhAA2K/OsCx3rnzu+g1PNNurRw=="}' 2024-09-16T15:52:32.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c43d1489-abce-4ac6-9678-ee6d2cca26cc -i td/osd-force-create-pg/2/new.json 2024-09-16T15:52:32.702 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:52:32.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/2/new.json 2024-09-16T15:52:32.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAVOhmTo4zBhAA2K/OsCx3rnzu+g1PNNurRw== --osd-uuid c43d1489-abce-4ac6-9678-ee6d2cca26cc 2024-09-16T15:52:32.753 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:32.754+0000 7fd6144f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:52:32.765 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:32.766+0000 7fd6144f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:52:32.766 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:32.766+0000 7fd6144f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:52:32.767 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:32.766+0000 7fd6144f77c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-09-16T15:52:32.767 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:32.766+0000 7fd6144f77c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-09-16T15:52:32.767 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:32.766+0000 7fd6144f77c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-09-16T15:52:32.849 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:32.846+0000 7fd6144f77c0 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2024-09-16T15:52:38.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2024-09-16T15:52:38.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:52:38.521 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T15:52:38.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T15:52:38.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:52:39.109 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:52:39.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T15:52:39.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:52:39.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:52:39.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:52:39.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:52:39.141 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:39.138+0000 7f6a6b77a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:52:39.153 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:39.154+0000 7f6a6b77a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:52:39.155 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:39.154+0000 7f6a6b77a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:52:39.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T15:52:39.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:52:39.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:52:39.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:52:39.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:52:39.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:39.567 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:52:39.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:52:39.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:39.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:39.718 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:39.718+0000 7f6a6b77a7c0 -1 Falling back to public interface 2024-09-16T15:52:39.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:40.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:40.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:40.998 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:52:40.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:52:40.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:40.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:41.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:42.264 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:52:42.262+0000 7f6a6b77a7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:52:42.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:42.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:42.498 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:52:42.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:52:42.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:42.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:43.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:44.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:44.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:44.054 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:52:44.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:52:44.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:44.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:44.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:45.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:45.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:45.512 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:52:45.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:52:45.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:45.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:46.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:47.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:47.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:47.095 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:52:47.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:52:47.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:47.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:47.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:48.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:48.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:48.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:52:48.526 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:52:48.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:48.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:49.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:50.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:50.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:50.088 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:52:50.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:52:50.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:50.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:50.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:51.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:51.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:51.513 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:52:51.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:52:51.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:51.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:52.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:53.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:53.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:53.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:52:53.031 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:52:53.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:53.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:53.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:54.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:54.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:54.589 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:52:54.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:52:54.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:54.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:55.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:56.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:56.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:56.006 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:52:56.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:52:56.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:56.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:56.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:57.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:57.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:57.571 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:52:57.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:52:57.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:57.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:52:58.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:52:59.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:52:59.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:52:59.620 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:52:59.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:52:59.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:52:59.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:53:00.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:01.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:01.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:01.346 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:53:01.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:53:01.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:01.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:53:02.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:03.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:03.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:03.403 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:53:03.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:53:03.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:03.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:53:04.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:05.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:05.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:05.248 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:53:05.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:53:05.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:05.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:53:05.692 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1402238249,v1:127.0.0.1:6819/1402238249] [v2:127.0.0.1:6820/1402238249,v1:127.0.0.1:6821/1402238249] exists,up c43d1489-abce-4ac6-9678-ee6d2cca26cc 2024-09-16T15:53:05.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:53:05.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:53:05.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:53:05.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: ceph osd pool create foo 50 2024-09-16T15:53:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T15:53:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:31: TEST_reuse_id: wait_for_clean 2024-09-16T15:53:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:53:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:53:06.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:53:06.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:53:06.462 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:53:06.462 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:53:06.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:53:06.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:53:06.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:53:06.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:53:06.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:53:06.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:53:06.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:53:06.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:53:06.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:53:07.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:53:07.091 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:53:07.091 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:53:07.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:53:07.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:53:07.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:53:07.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836497 2024-09-16T15:53:07.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836497 2024-09-16T15:53:07.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836497' 2024-09-16T15:53:07.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:53:07.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:53:07.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672979 2024-09-16T15:53:07.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672979 2024-09-16T15:53:07.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836497 1-42949672979' 2024-09-16T15:53:07.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:53:07.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:53:07.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509448 2024-09-16T15:53:07.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509448 2024-09-16T15:53:07.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836497 1-42949672979 2-64424509448' 2024-09-16T15:53:07.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:53:07.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836497 2024-09-16T15:53:07.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:53:07.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:53:07.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836497 2024-09-16T15:53:07.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:53:07.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836497 2024-09-16T15:53:07.829 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836497 2024-09-16T15:53:07.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-09-16T15:53:07.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:53:08.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836495 -lt 21474836497 2024-09-16T15:53:08.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:53:09.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:53:09.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:53:09.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836498 -lt 21474836497 2024-09-16T15:53:09.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:53:09.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672979 2024-09-16T15:53:09.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:53:09.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:53:09.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672979 2024-09-16T15:53:09.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:53:09.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672979 2024-09-16T15:53:09.868 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672979 2024-09-16T15:53:09.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-16T15:53:09.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:53:10.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-16T15:53:10.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:53:10.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:53:10.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509448 2024-09-16T15:53:10.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:53:10.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:53:10.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509448 2024-09-16T15:53:10.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509448 2024-09-16T15:53:10.318 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509448 2024-09-16T15:53:10.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-16T15:53:10.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:53:10.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-16T15:53:10.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:53:10.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:53:10.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:53:11.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-09-16T15:53:11.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:53:11.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:53:11.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:53:11.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:53:11.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:53:11.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:53:11.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:53:11.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-09-16T15:53:11.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:53:11.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:53:11.708 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:53:12.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-09-16T15:53:12.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:53:12.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:53:12.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2024-09-16T15:53:12.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:53:12.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:53:12.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:53:12.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:53:12.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:53:12.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:53:12.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2024-09-16T15:53:12.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:53:12.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:53:12.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:53:12.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:53:12.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:53:12.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:53:12.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2024-09-16T15:53:12.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:53:12.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:53:12.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:53:12.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:53:12.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:53:13.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:53:13.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2024-09-16T15:53:14.283 INFO:tasks.workunit.client.0.smithi039.stderr:PG '1.0' not found 2024-09-16T15:53:14.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2024-09-16T15:53:15.879 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-16T15:53:15.879 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-16T15:53:15.879 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-16T15:53:15.953 INFO:tasks.workunit.client.0.smithi039.stdout:Remove successful 2024-09-16T15:53:16.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2024-09-16T15:53:17.634 INFO:tasks.workunit.client.0.smithi039.stderr:PG '1.0' not found 2024-09-16T15:53:18.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2024-09-16T15:53:18.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-09-16T15:53:18.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/0 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-16T15:53:18.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:53:18.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:53:18.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:53:18.175 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:53:18.175 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:18.175 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:53:18.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:53:18.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:53:18.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:53:18.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-16T15:53:18.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-16T15:53:18.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:53:18.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:53:18.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:53:18.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:53:18.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:53:18.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:53:18.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/0 2024-09-16T15:53:18.176 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:53:18.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T15:53:18.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:53:18.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/0/whoami 2024-09-16T15:53:18.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T15:53:18.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:53:18.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:53:18.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:53:18.213 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:18.213+0000 7f461a05e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:18.217 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:18.213+0000 7f461a05e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:18.218 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:18.217+0000 7f461a05e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:18.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T15:53:18.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:53:18.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:53:18.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:53:18.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:53:18.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:18.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:53:18.631 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:53:18.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:18.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:53:18.766 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:18.765+0000 7f461a05e7c0 -1 Falling back to public interface 2024-09-16T15:53:19.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:20.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:20.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:20.105 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:53:20.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:53:20.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:20.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:53:20.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:21.390 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:21.389+0000 7f461a05e7c0 -1 osd.0 23 log_to_monitors true 2024-09-16T15:53:21.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:21.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:21.519 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:53:21.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:53:21.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:21.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:53:21.937 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 28 up_thru 19 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/97051380,v1:127.0.0.1:6803/97051380] [v2:127.0.0.1:6804/97051380,v1:127.0.0.1:6805/97051380] exists,up c1d5ce29-cf35-4185-b2ff-cfb3f6b74701 2024-09-16T15:53:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:53:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:53:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:53:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2024-09-16T15:53:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-09-16T15:53:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:53:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T15:53:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:53:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/1 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/1 2024-09-16T15:53:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T15:53:21.943 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:53:21.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:53:21.943 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/1/whoami 2024-09-16T15:53:21.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T15:53:21.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:53:21.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:53:21.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:53:21.970 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:21.969+0000 7f30a4ebf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:22.029 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:22.029+0000 7f30a4ebf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:22.031 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:22.029+0000 7f30a4ebf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:22.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T15:53:22.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:53:22.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:53:22.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:53:22.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:53:22.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:22.355 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:53:22.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:53:22.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:22.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:53:22.570 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:22.569+0000 7f30a4ebf7c0 -1 Falling back to public interface 2024-09-16T15:53:22.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:23.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:23.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:23.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:53:23.800 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:53:23.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:23.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:53:24.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:25.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:25.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:25.255 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:53:25.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:53:25.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:25.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:53:25.293 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:25.289+0000 7f30a4ebf7c0 -1 osd.1 24 log_to_monitors true 2024-09-16T15:53:25.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:26.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:26.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:26.731 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:53:26.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:53:26.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:26.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:53:27.158 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 31 up_thru 31 down_at 25 last_clean_interval [10,24) [v2:127.0.0.1:6810/2187259382,v1:127.0.0.1:6811/2187259382] [v2:127.0.0.1:6812/2187259382,v1:127.0.0.1:6813/2187259382] exists,up 7c235452-bddc-4c87-9b02-9f2d120bab43 2024-09-16T15:53:27.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:53:27.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/2 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-09-16T15:53:27.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T15:53:27.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T15:53:27.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-16T15:53:27.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T15:53:27.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:53:27.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:53:27.162 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:53:27.162 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:27.162 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:53:27.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:53:27.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' 2024-09-16T15:53:27.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:53:27.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-16T15:53:27.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-16T15:53:27.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:53:27.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:53:27.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:53:27.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:53:27.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T15:53:27.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T15:53:27.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/2 2024-09-16T15:53:27.164 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:53:27.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T15:53:27.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=a71f2aaf-bdf0-46bf-ac18-9f4596888e72 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:53:27.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/2/whoami 2024-09-16T15:53:27.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T15:53:27.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T15:53:27.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T15:53:27.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T15:53:27.197 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:27.197+0000 7f89773667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:27.232 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:27.229+0000 7f89773667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:27.233 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:27.233+0000 7f89773667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:27.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T15:53:27.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:53:27.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:53:27.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:53:27.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:53:27.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:27.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:53:27.645 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:53:27.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:27.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:53:27.794 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:27.793+0000 7f89773667c0 -1 Falling back to public interface 2024-09-16T15:53:28.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:29.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:29.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:29.125 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:53:29.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:53:29.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:29.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:53:29.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:30.269 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:30.269+0000 7f89773667c0 -1 osd.2 25 log_to_monitors true 2024-09-16T15:53:30.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:30.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:30.554 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:53:30.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:53:30.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:30.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:53:31.199 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 34 up_thru 34 down_at 26 last_clean_interval [15,25) [v2:127.0.0.1:6818/1438284720,v1:127.0.0.1:6819/1438284720] [v2:127.0.0.1:6820/1438284720,v1:127.0.0.1:6821/1438284720] exists,up c43d1489-abce-4ac6-9678-ee6d2cca26cc 2024-09-16T15:53:31.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:53:31.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:53:31.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:53:31.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: sleep 10 2024-09-16T15:53:41.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: ceph pg ls 2024-09-16T15:53:41.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep 1.0 2024-09-16T15:53:41.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep stale 2024-09-16T15:53:41.622 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 stale+active+clean 34s 0'0 22:12 [1]p1 [1]p1 2024-09-16T15:53:06.063210+0000 2024-09-16T15:53:06.063210+0000 0 periodic scrub scheduled @ 2024-09-17T16:57:30.044607+0000 2024-09-16T15:53:41.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2024-09-16T15:53:42.322 INFO:tasks.workunit.client.0.smithi039.stderr:pg 1.0 already creating 2024-09-16T15:53:42.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:46: TEST_reuse_id: wait_for_clean 2024-09-16T15:53:42.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T15:53:42.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T15:53:42.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T15:53:42.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T15:53:42.347 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T15:53:42.347 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T15:53:42.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T15:53:42.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T15:53:42.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T15:53:42.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T15:53:42.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T15:53:42.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T15:53:42.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T15:53:42.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T15:53:42.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T15:53:42.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T15:53:42.903 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T15:53:42.903 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T15:53:42.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T15:53:42.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:53:42.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T15:53:43.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084295 2024-09-16T15:53:43.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084295 2024-09-16T15:53:43.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-120259084295' 2024-09-16T15:53:43.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:53:43.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T15:53:43.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986182 2024-09-16T15:53:43.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986182 2024-09-16T15:53:43.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-120259084295 1-133143986182' 2024-09-16T15:53:43.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T15:53:43.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T15:53:43.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=146028888069 2024-09-16T15:53:43.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 146028888069 2024-09-16T15:53:43.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-120259084295 1-133143986182 2-146028888069' 2024-09-16T15:53:43.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:53:43.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-120259084295 2024-09-16T15:53:43.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:53:43.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T15:53:43.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-120259084295 2024-09-16T15:53:43.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:53:43.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084295 2024-09-16T15:53:43.381 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 120259084295 2024-09-16T15:53:43.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 120259084295' 2024-09-16T15:53:43.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:53:43.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084293 -lt 120259084295 2024-09-16T15:53:43.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T15:53:44.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T15:53:44.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T15:53:45.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084295 -lt 120259084295 2024-09-16T15:53:45.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:53:45.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-133143986182 2024-09-16T15:53:45.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:53:45.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T15:53:45.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-133143986182 2024-09-16T15:53:45.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:53:45.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986182 2024-09-16T15:53:45.289 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 133143986182 2024-09-16T15:53:45.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 133143986182' 2024-09-16T15:53:45.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T15:53:45.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986182 -lt 133143986182 2024-09-16T15:53:45.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T15:53:45.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-146028888069 2024-09-16T15:53:45.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T15:53:45.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T15:53:45.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T15:53:45.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-146028888069 2024-09-16T15:53:45.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=146028888069 2024-09-16T15:53:45.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 146028888069' 2024-09-16T15:53:45.724 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 146028888069 2024-09-16T15:53:45.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T15:53:46.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 146028888069 -lt 146028888069 2024-09-16T15:53:46.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T15:53:46.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:53:46.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:53:46.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-09-16T15:53:46.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T15:53:46.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T15:53:46.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T15:53:46.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T15:53:46.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T15:53:46.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T15:53:46.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T15:53:47.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-09-16T15:53:47.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T15:53:47.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T15:53:47.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T15:53:47.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-09-16T15:53:47.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T15:53:47.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T15:53:47.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:17: run: teardown td/osd-force-create-pg 2024-09-16T15:53:47.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-09-16T15:53:47.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:53:47.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-09-16T15:53:47.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:53:47.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:53:47.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:53:47.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:53:47.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:53:47.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:53:47.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:53:47.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:53:47.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:53:47.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:53:47.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:53:47.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:53:47.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:53:47.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:53:47.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:53:47.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:53:47.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:53:47.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:53:47.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-09-16T15:53:47.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:53:47.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:47.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:53:47.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168548 2024-09-16T15:53:47.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:53:47.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:53:47.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T15:53:47.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-force-create-pg 0 2024-09-16T15:53:47.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-09-16T15:53:47.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T15:53:47.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-09-16T15:53:47.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:53:47.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:53:47.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:53:47.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:53:47.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:53:47.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:53:47.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:53:47.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:53:47.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:53:47.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:53:47.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:53:47.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:53:47.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:53:47.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:53:47.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:53:47.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:53:47.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:53:47.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T15:53:47.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-09-16T15:53:47.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:53:47.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:47.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168548 2024-09-16T15:53:47.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168548 2024-09-16T15:53:47.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:53:47.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:53:47.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T15:53:47.974 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T15:53:47.974 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T15:53:48.028 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2024-09-16T15:53:48.029 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2024-09-16T15:53:48.081 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T15:53:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T15:53:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:53:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T15:53:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T15:53:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T15:53:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T15:53:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T15:53:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T15:53:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T15:53:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-markdown 2024-09-16T15:53:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2024-09-16T15:53:48.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2024-09-16T15:53:48.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2024-09-16T15:53:48.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2024-09-16T15:53:48.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2024-09-16T15:53:48.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2024-09-16T15:53:48.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none ' 2024-09-16T15:53:48.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2024-09-16T15:53:48.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2024-09-16T15:53:48.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T15:53:48.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2024-09-16T15:53:48.091 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_markdown_boot_exceed_time 2024-09-16T15:53:48.091 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_markdown_exceed_maxdown_count 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_osd_stop' 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:53:48.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:53:48.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:53:48.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:53:48.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:53:48.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:53:48.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:53:48.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:53:48.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:53:48.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:53:48.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:53:48.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:53:48.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:53:48.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:53:48.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:53:48.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-16T15:53:48.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:53:48.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:48.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:53:48.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.177538 2024-09-16T15:53:48.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:53:48.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:53:48.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-16T15:53:48.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:53:48.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:48.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:53:48.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.177538 2024-09-16T15:53:48.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:53:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T15:53:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T15:53:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:53:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-16T15:53:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2024-09-16T15:53:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2024-09-16T15:53:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2024-09-16T15:53:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-16T15:53:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:53:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:53:48.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:53:48.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-16T15:53:48.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-16T15:53:48.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:53:48.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:53:48.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:53:48.496 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:53:48.496 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:48.496 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:53:48.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:53:48.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:53:48.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:53:48.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:53:48.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:53:48.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:53:48.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:53:48.701 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:53:48.702 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:53:48.702 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:53:48.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:53:48.702 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:53:48.702 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:48.702 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:53:48.702 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.177538/ceph-mon.a.asok 2024-09-16T15:53:48.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:53:48.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.177538/ceph-mon.a.asok config get fsid 2024-09-16T15:53:48.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:53:48.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:53:48.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:53:48.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:53:48.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:53:48.819 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:53:48.819 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:53:48.819 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:53:48.820 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:53:48.820 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:48.820 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:53:48.820 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.177538/ceph-mon.a.asok 2024-09-16T15:53:48.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:53:48.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.177538/ceph-mon.a.asok config get mon_host 2024-09-16T15:53:48.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2024-09-16T15:53:48.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-16T15:53:48.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:53:48.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:53:48.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:53:48.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-16T15:53:48.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:53:49.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:53:49.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:53:49.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:53:49.466 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:53:49.466 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:49.466 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:53:49.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:53:49.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:53:49.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:53:49.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2024-09-16T15:53:49.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T15:53:49.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:53:49.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:53:49.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:53:49.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:53:49.504 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:53:49.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:53:49.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-16T15:53:49.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:53:49.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa5ad751-3fb9-43f4-bfca-c98f33c9bcec 2024-09-16T15:53:49.506 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 aa5ad751-3fb9-43f4-bfca-c98f33c9bcec 2024-09-16T15:53:49.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 aa5ad751-3fb9-43f4-bfca-c98f33c9bcec' 2024-09-16T15:53:49.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:53:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNVOhmY1YgHxAAcAyDRqNjBieZxCK+M5h8oA== 2024-09-16T15:53:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNVOhmY1YgHxAAcAyDRqNjBieZxCK+M5h8oA=="}' 2024-09-16T15:53:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa5ad751-3fb9-43f4-bfca-c98f33c9bcec -i td/osd-markdown/0/new.json 2024-09-16T15:53:49.880 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:53:49.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-16T15:53:49.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNVOhmY1YgHxAAcAyDRqNjBieZxCK+M5h8oA== --osd-uuid aa5ad751-3fb9-43f4-bfca-c98f33c9bcec 2024-09-16T15:53:49.930 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:49.929+0000 7fc18fb417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:49.932 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:49.929+0000 7fc18fb417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:49.933 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:49.933+0000 7fc18fb417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:49.933 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:49.933+0000 7fc18fb417c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T15:53:49.934 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:49.933+0000 7fc18fb417c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T15:53:49.934 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:49.933+0000 7fc18fb417c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T15:53:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:50.009+0000 7fc18fb417c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-16T15:53:56.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-16T15:53:56.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:53:56.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:53:56.007 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:53:56.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:53:56.565 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:53:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:53:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:53:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:53:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:53:56.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:53:56.595 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:56.593+0000 7fc027b987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:56.620 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:56.617+0000 7fc027b987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:56.622 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:56.621+0000 7fc027b987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:53:57.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:53:57.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:53:57.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:53:57.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:53:57.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:53:57.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:57.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:53:57.033 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:53:57.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:57.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:53:57.190 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:57.189+0000 7fc027b987c0 -1 Falling back to public interface 2024-09-16T15:53:57.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:58.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:58.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:58.519 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:53:58.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:53:58.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:58.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:53:58.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:53:59.764 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:53:59.760+0000 7fc027b987c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:53:59.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:53:59.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:53:59.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:53:59.933 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:53:59.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:53:59.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:00.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:01.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:01.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:01.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:54:01.464 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:54:01.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:01.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:01.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:02.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:02.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:02.932 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:54:02.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:54:02.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:02.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:03.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:04.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:04.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:04.502 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:54:04.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:54:04.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:04.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:04.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:05.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:05.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:05.927 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:54:05.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:54:05.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:05.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:06.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:07.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:07.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:07.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:54:07.460 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:54:07.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:07.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:07.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:08.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:08.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:08.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:54:08.889 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:54:08.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:08.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:09.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:10.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:10.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:10.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:54:10.319 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:54:10.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:10.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:10.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:11.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:11.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:11.815 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:54:11.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:54:11.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:11.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:12.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:13.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:13.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:13.413 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:54:13.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:54:13.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:13.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:13.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:14.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:14.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:14.974 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:54:14.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:54:14.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:14.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:15.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:16.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:16.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:16.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:54:16.789 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:54:16.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:16.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:17.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:18.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:18.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:18.421 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:54:18.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:54:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:19.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:20.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:20.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:20.746 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:54:20.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:54:20.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:20.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:21.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:22.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:22.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:22.678 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:54:22.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:54:22.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:22.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:54:23.106 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/721799030,v1:127.0.0.1:6803/721799030] [v2:127.0.0.1:6804/721799030,v1:127.0.0.1:6805/721799030] exists,up aa5ad751-3fb9-43f4-bfca-c98f33c9bcec 2024-09-16T15:54:23.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-16T15:54:23.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:54:23.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T15:54:23.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T15:54:23.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:54:23.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:54:23.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:54:23.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:54:23.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:54:23.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:54:23.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-16T15:54:23.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:54:23.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9d39744d-4686-4ca0-b223-88afabb0a8e9 2024-09-16T15:54:23.113 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 9d39744d-4686-4ca0-b223-88afabb0a8e9 2024-09-16T15:54:23.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9d39744d-4686-4ca0-b223-88afabb0a8e9' 2024-09-16T15:54:23.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:54:23.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvVOhmrJyyBxAAxv+0mT2UL3C0qEgWN7We4w== 2024-09-16T15:54:23.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvVOhmrJyyBxAAxv+0mT2UL3C0qEgWN7We4w=="}' 2024-09-16T15:54:23.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9d39744d-4686-4ca0-b223-88afabb0a8e9 -i td/osd-markdown/1/new.json 2024-09-16T15:54:23.675 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:54:23.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-16T15:54:23.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvVOhmrJyyBxAAxv+0mT2UL3C0qEgWN7We4w== --osd-uuid 9d39744d-4686-4ca0-b223-88afabb0a8e9 2024-09-16T15:54:23.728 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:23.724+0000 7f395e0777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:54:23.756 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:23.752+0000 7f395e0777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:54:23.758 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:23.756+0000 7f395e0777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:54:23.758 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:23.756+0000 7f395e0777c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T15:54:23.758 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:23.756+0000 7f395e0777c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T15:54:23.758 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:23.756+0000 7f395e0777c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T15:54:23.839 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:23.836+0000 7f395e0777c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-16T15:54:29.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-16T15:54:29.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:54:29.685 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:54:29.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:54:29.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:54:30.241 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:54:30.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:54:30.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:54:30.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:54:30.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:54:30.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:54:30.272 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:30.268+0000 7fbf736bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:54:30.286 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:30.284+0000 7fbf736bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:54:30.287 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:30.284+0000 7fbf736bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:54:30.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:54:30.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:54:30.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:54:30.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:54:30.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:54:30.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:30.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:54:30.710 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:54:30.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:30.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:30.858 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:30.856+0000 7fbf736bd7c0 -1 Falling back to public interface 2024-09-16T15:54:31.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:32.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:32.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:32.126 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:54:32.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:54:32.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:32.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:32.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:33.363 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:33.360+0000 7fbf736bd7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:54:33.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:33.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:33.578 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:54:33.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:54:33.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:33.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:34.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:35.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:35.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:35.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:54:35.184 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:54:35.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:35.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:35.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:36.668 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:54:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:54:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:36.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:37.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:38.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:38.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:38.222 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:54:38.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:54:38.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:38.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:38.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:39.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:39.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:39.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:54:39.834 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:54:39.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:39.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:40.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:41.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:54:41.328 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:54:41.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:41.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:41.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:42.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:42.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:42.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:54:42.812 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:54:42.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:42.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:43.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:44.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:44.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:44.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:54:44.274 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:54:44.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:44.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:44.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:45.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:45.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:45.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:54:45.819 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:54:45.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:45.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:54:47.769 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:54:47.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:47.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:48.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:49.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:49.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:49.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:54:49.530 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:54:49.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:49.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:50.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:51.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:51.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:51.346 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:54:51.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:54:51.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:51.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:52.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:53.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:53.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:53.379 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:54:53.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:54:53.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:53.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:54.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:55.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:55.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:55.350 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:54:55.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:54:55.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:55.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:56.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:54:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:54:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:54:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:54:57.060 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:54:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:54:57.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:54:57.481 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1199774919,v1:127.0.0.1:6811/1199774919] [v2:127.0.0.1:6812/1199774919,v1:127.0.0.1:6813/1199774919] exists,up 9d39744d-4686-4ca0-b223-88afabb0a8e9 2024-09-16T15:54:57.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:54:57.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:54:57.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:54:57.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2024-09-16T15:54:57.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T15:54:57.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:54:57.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T15:54:57.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:54:57.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:54:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:54:57.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:54:57.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-16T15:54:57.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:54:57.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=db383b45-a1d5-4e18-a809-a41195d24fc7 2024-09-16T15:54:57.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 db383b45-a1d5-4e18-a809-a41195d24fc7' 2024-09-16T15:54:57.488 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 db383b45-a1d5-4e18-a809-a41195d24fc7 2024-09-16T15:54:57.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:54:57.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRVOhmZ85PHhAAnoeRsdiFv1aHf8w+VRXNaw== 2024-09-16T15:54:57.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRVOhmZ85PHhAAnoeRsdiFv1aHf8w+VRXNaw=="}' 2024-09-16T15:54:57.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new db383b45-a1d5-4e18-a809-a41195d24fc7 -i td/osd-markdown/2/new.json 2024-09-16T15:54:58.133 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:54:58.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-16T15:54:58.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRVOhmZ85PHhAAnoeRsdiFv1aHf8w+VRXNaw== --osd-uuid db383b45-a1d5-4e18-a809-a41195d24fc7 2024-09-16T15:54:58.184 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:58.179+0000 7ff21ff197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:54:58.197 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:58.191+0000 7ff21ff197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:54:58.198 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:58.195+0000 7ff21ff197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:54:58.199 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:58.195+0000 7ff21ff197c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T15:54:58.199 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:58.195+0000 7ff21ff197c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T15:54:58.199 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:58.195+0000 7ff21ff197c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T15:54:58.289 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:54:58.287+0000 7ff21ff197c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-16T15:55:04.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-16T15:55:04.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:55:04.085 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T15:55:04.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T15:55:04.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:55:04.644 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:55:04.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T15:55:04.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:55:04.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:55:04.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:55:04.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:55:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:55:04.671+0000 7f3aef9507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:55:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:55:04.675+0000 7f3aef9507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:55:04.679 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:55:04.675+0000 7f3aef9507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:55:05.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T15:55:05.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:55:05.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:55:05.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:55:05.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:55:05.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:05.070 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:55:05.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:55:05.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:05.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:05.246 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:55:05.243+0000 7f3aef9507c0 -1 Falling back to public interface 2024-09-16T15:55:05.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:06.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:06.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:06.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:55:06.488 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:55:06.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:06.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:06.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:07.781 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:55:07.775+0000 7f3aef9507c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:55:07.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:07.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:07.900 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:55:07.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:55:07.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:07.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:09.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:09.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:09.501 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:55:09.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:55:09.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:09.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:09.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:10.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:10.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:10.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:55:10.931 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:55:10.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:10.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:11.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:12.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:12.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:12.597 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:55:12.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:55:12.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:12.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:13.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:14.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:14.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:14.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:55:14.074 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:55:14.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:14.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:14.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:15.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:15.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:15.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:55:15.613 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:55:15.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:15.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:16.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:17.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:17.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:17.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:55:17.036 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:55:17.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:17.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:17.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:18.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:18.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:18.540 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:55:18.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:55:18.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:18.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:19.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:20.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:20.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:20.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:55:20.004 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:55:20.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:20.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:20.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:21.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:21.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:21.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:55:21.493 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:55:21.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:21.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:22.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:23.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:23.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:23.554 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:55:23.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:55:23.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:23.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:24.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:25.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:25.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:25.612 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:55:25.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:55:25.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:25.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:26.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:27.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:27.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:27.381 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:55:27.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:55:27.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:27.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:28.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:29.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:29.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:29.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:55:29.123 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:55:29.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:29.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:30.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:55:31.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:55:31.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:55:31.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:55:31.096 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:55:31.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:55:31.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:55:31.524 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/769374553,v1:127.0.0.1:6819/769374553] [v2:127.0.0.1:6820/769374553,v1:127.0.0.1:6821/769374553] exists,up db383b45-a1d5-4e18-a809-a41195d24fc7 2024-09-16T15:55:31.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:55:31.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:55:31.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:55:31.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2024-09-16T15:55:31.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T15:55:31.973 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-09-16T15:55:31.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T15:55:31.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T15:55:32.621 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' already exists 2024-09-16T15:55:32.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:55:33.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T15:55:35.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2024-09-16T15:55:35.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2024-09-16T15:55:35.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2024-09-16T15:55:35.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-09-16T15:55:35.687 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T15:55:35.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2024-09-16T15:55:35.838 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T15:55:35.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2024-09-16T15:55:35.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2024-09-16T15:55:35.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2024-09-16T15:55:35.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-09-16T15:55:35.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2024-09-16T15:55:35.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-16T15:55:35.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-16T15:55:36.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-16T15:55:36.410 INFO:tasks.workunit.client.0.smithi039.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T15:55:36.410 INFO:tasks.workunit.client.0.smithi039.stdout:-1 0.29306 root default 2024-09-16T15:55:36.410 INFO:tasks.workunit.client.0.smithi039.stdout:-3 0.29306 host smithi039 2024-09-16T15:55:36.410 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:55:36.410 INFO:tasks.workunit.client.0.smithi039.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-16T15:55:36.410 INFO:tasks.workunit.client.0.smithi039.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-16T15:55:36.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-16T15:55:36.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-16T15:55:36.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-16T15:55:36.885 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:55:36.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-16T15:55:36.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-16T15:55:37.333 INFO:tasks.workunit.client.0.smithi039.stderr:marked down osd.0. 2024-09-16T15:55:37.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-16T15:55:47.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-16T15:55:47.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-16T15:55:47.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-16T15:55:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T15:55:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:-1 0.29306 root default 2024-09-16T15:55:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:-3 0.29306 host smithi039 2024-09-16T15:55:47.969 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:55:47.969 INFO:tasks.workunit.client.0.smithi039.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-16T15:55:47.969 INFO:tasks.workunit.client.0.smithi039.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-16T15:55:47.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-16T15:55:47.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-16T15:55:47.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-16T15:55:48.434 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:55:48.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-16T15:55:48.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-16T15:55:48.909 INFO:tasks.workunit.client.0.smithi039.stderr:marked down osd.0. 2024-09-16T15:55:48.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-16T15:55:58.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-16T15:55:58.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-16T15:55:59.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-16T15:55:59.508 INFO:tasks.workunit.client.0.smithi039.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T15:55:59.508 INFO:tasks.workunit.client.0.smithi039.stdout:-1 0.29306 root default 2024-09-16T15:55:59.508 INFO:tasks.workunit.client.0.smithi039.stdout:-3 0.29306 host smithi039 2024-09-16T15:55:59.508 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:55:59.508 INFO:tasks.workunit.client.0.smithi039.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-16T15:55:59.508 INFO:tasks.workunit.client.0.smithi039.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-16T15:55:59.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-16T15:55:59.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-16T15:55:59.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-16T15:55:59.992 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:55:59.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-16T15:55:59.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-16T15:56:00.552 INFO:tasks.workunit.client.0.smithi039.stderr:marked down osd.0. 2024-09-16T15:56:00.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-16T15:56:10.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2024-09-16T15:56:25.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2024-09-16T15:56:25.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2024-09-16T15:56:25.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2024-09-16T15:56:25.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2024-09-16T15:56:26.189 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:56:26.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-16T15:56:26.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-16T15:56:26.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:56:26.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-16T15:56:26.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:56:26.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:56:26.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:56:26.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:56:26.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:56:26.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:56:26.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:56:26.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:56:26.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:56:26.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:56:26.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:56:26.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:56:26.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:56:26.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:56:26.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:56:26.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:56:26.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:56:26.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:56:26.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-16T15:56:26.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:56:26.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:56:26.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:56:26.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.177538 2024-09-16T15:56:26.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:56:26.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:56:26.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-16T15:56:26.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-16T15:56:26.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-16T15:56:26.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-16T15:56:26.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-16T15:56:26.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:56:26.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-16T15:56:26.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:56:26.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:56:26.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:56:26.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:56:26.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:56:26.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:56:26.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:56:26.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:56:26.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:56:26.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:56:26.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:56:26.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:56:26.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:56:26.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:56:26.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:56:26.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:56:26.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:56:26.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:56:26.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-16T15:56:26.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:56:26.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:56:26.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:56:26.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.177538 2024-09-16T15:56:26.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:56:26.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:56:26.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-16T15:56:26.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:56:26.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:56:26.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:56:26.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.177538 2024-09-16T15:56:26.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:56:26.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T15:56:26.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:56:26.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-16T15:56:26.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2024-09-16T15:56:26.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2024-09-16T15:56:26.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2024-09-16T15:56:26.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-16T15:56:26.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:56:26.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:56:26.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:56:26.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-16T15:56:26.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-16T15:56:26.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:56:26.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:56:26.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:56:26.659 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:56:26.659 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:56:26.659 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:56:26.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:56:26.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:56:26.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:56:26.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:56:26.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:56:26.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:56:26.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:56:26.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:56:26.895 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:56:26.895 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:56:26.895 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:56:26.896 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:56:26.896 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:56:26.896 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:56:26.896 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.177538/ceph-mon.a.asok 2024-09-16T15:56:26.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:56:26.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.177538/ceph-mon.a.asok config get fsid 2024-09-16T15:56:27.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:56:27.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:56:27.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:56:27.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:56:27.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:56:27.015 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:56:27.015 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:56:27.015 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:56:27.016 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:56:27.016 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:56:27.016 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:56:27.016 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.177538/ceph-mon.a.asok 2024-09-16T15:56:27.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:56:27.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.177538/ceph-mon.a.asok config get mon_host 2024-09-16T15:56:27.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2024-09-16T15:56:27.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-16T15:56:27.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:56:27.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:56:27.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:56:27.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-16T15:56:27.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:56:28.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:56:28.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:56:28.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:56:28.052 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:56:28.052 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:56:28.052 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:56:28.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:56:28.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:56:28.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-16T15:56:28.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:56:28.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:56:28.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T15:56:28.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:56:28.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:56:28.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:56:28.117 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:56:28.117 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:56:28.117 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:56:28.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:56:28.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:56:28.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:56:28.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:56:28.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:56:28.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T15:56:28.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T15:56:28.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:56:28.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:56:28.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:56:28.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:56:28.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:56:28.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:56:28.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-16T15:56:28.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:56:28.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e911117-36b2-40f3-b857-64fd7928aa21 2024-09-16T15:56:28.120 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 9e911117-36b2-40f3-b857-64fd7928aa21 2024-09-16T15:56:28.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9e911117-36b2-40f3-b857-64fd7928aa21' 2024-09-16T15:56:28.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:56:28.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsVehmNKwmCBAAVXT4CeGiKXcNxfzC7JbozA== 2024-09-16T15:56:28.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsVehmNKwmCBAAVXT4CeGiKXcNxfzC7JbozA=="}' 2024-09-16T15:56:28.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e911117-36b2-40f3-b857-64fd7928aa21 -i td/osd-markdown/0/new.json 2024-09-16T15:56:28.490 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:56:28.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-16T15:56:28.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsVehmNKwmCBAAVXT4CeGiKXcNxfzC7JbozA== --osd-uuid 9e911117-36b2-40f3-b857-64fd7928aa21 2024-09-16T15:56:28.537 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:28.534+0000 7f77459e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:56:28.539 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:28.538+0000 7f77459e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:56:28.540 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:28.538+0000 7f77459e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:56:28.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:28.538+0000 7f77459e67c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T15:56:28.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:28.538+0000 7f77459e67c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T15:56:28.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:28.538+0000 7f77459e67c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T15:56:28.613 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:28.614+0000 7f77459e67c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-16T15:56:34.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-16T15:56:34.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:56:34.547 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:56:34.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:56:34.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:56:35.107 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:56:35.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:56:35.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:56:35.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:56:35.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:56:35.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:56:35.137 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:35.137+0000 7fb17f3ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:56:35.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:35.161+0000 7fb17f3ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:56:35.163 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:35.161+0000 7fb17f3ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:56:35.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:56:35.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:56:35.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:56:35.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:56:35.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:56:35.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:35.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:56:35.527 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:56:35.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:35.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:35.719 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:35.717+0000 7fb17f3ab7c0 -1 Falling back to public interface 2024-09-16T15:56:35.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:36.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:36.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:36.940 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:56:36.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:56:36.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:36.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:37.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:38.262 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:56:38.261+0000 7fb17f3ab7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:56:38.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:38.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:38.422 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:56:38.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:56:38.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:38.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:39.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:40.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:40.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:40.268 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:56:40.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:56:40.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:40.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:40.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:41.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:41.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:41.706 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:56:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:56:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:42.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:43.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:43.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:43.306 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:56:43.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:56:43.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:43.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:43.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:44.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:44.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:44.747 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:56:44.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:56:44.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:44.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:45.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:46.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:46.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:46.342 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:56:46.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:56:46.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:46.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:47.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:47.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:47.771 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:56:47.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:56:47.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:47.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:48.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:49.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:49.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:49.295 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:56:49.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:56:49.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:49.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:49.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:50.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:50.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:50.730 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:56:50.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:56:50.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:50.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:51.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:52.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:52.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:52.585 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:56:52.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:56:52.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:52.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:53.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:54.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:54.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:54.033 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:56:54.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:56:54.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:54.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:54.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:55.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:55.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:55.595 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:56:55.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:56:55.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:55.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:56.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:57.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:57.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:57.169 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:56:57.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:56:57.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:57.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:56:58.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:56:59.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:56:59.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:56:59.175 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:56:59.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:56:59.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:56:59.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:57:00.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:01.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:01.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:01.347 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:57:01.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:57:01.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:01.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:57:01.834 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4273701055,v1:127.0.0.1:6803/4273701055] [v2:127.0.0.1:6804/4273701055,v1:127.0.0.1:6805/4273701055] exists,up 9e911117-36b2-40f3-b857-64fd7928aa21 2024-09-16T15:57:01.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:57:01.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:57:01.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:57:01.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2024-09-16T15:57:01.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T15:57:01.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:57:01.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:57:01.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:57:01.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-16T15:57:01.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T15:57:01.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:57:01.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:57:01.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:57:01.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-16T15:57:01.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-16T15:57:01.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:57:01.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:57:01.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T15:57:01.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:57:01.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:57:01.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:57:01.840 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:57:01.840 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:57:01.840 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:57:01.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:57:01.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:57:01.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:57:01.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:57:01.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:57:01.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T15:57:01.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T15:57:01.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:57:01.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:57:01.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:57:01.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:57:01.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:57:01.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:57:01.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-16T15:57:01.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:57:01.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ff7c030-7d31-4d66-8456-36f599a7dac3 2024-09-16T15:57:01.842 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 8ff7c030-7d31-4d66-8456-36f599a7dac3 2024-09-16T15:57:01.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8ff7c030-7d31-4d66-8456-36f599a7dac3' 2024-09-16T15:57:01.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:57:01.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNVehmhQYMMxAA/yyq6647ts4Y83CwfMDehA== 2024-09-16T15:57:01.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNVehmhQYMMxAA/yyq6647ts4Y83CwfMDehA=="}' 2024-09-16T15:57:01.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ff7c030-7d31-4d66-8456-36f599a7dac3 -i td/osd-markdown/1/new.json 2024-09-16T15:57:02.395 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:57:02.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-16T15:57:02.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNVehmhQYMMxAA/yyq6647ts4Y83CwfMDehA== --osd-uuid 8ff7c030-7d31-4d66-8456-36f599a7dac3 2024-09-16T15:57:02.440 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:02.437+0000 7fe7b0cf27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:02.445 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:02.441+0000 7fe7b0cf27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:02.445 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:02.441+0000 7fe7b0cf27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:02.445 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:02.441+0000 7fe7b0cf27c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T15:57:02.445 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:02.441+0000 7fe7b0cf27c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T15:57:02.445 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:02.441+0000 7fe7b0cf27c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T15:57:02.534 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:02.533+0000 7fe7b0cf27c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-16T15:57:08.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-16T15:57:08.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:57:08.588 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:57:08.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:57:08.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:57:09.177 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:57:09.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:57:09.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:57:09.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:57:09.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:57:09.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:57:09.208 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:09.205+0000 7fe65b3ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:09.210 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:09.209+0000 7fe65b3ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:09.211 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:09.209+0000 7fe65b3ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:09.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:57:09.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:57:09.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:57:09.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:57:09.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:57:09.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:09.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:57:09.666 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:57:09.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:09.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:09.794 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:09.793+0000 7fe65b3ac7c0 -1 Falling back to public interface 2024-09-16T15:57:10.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:11.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:11.125 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:57:11.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:11.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:57:11.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:11.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:11.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:12.285 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:12.285+0000 7fe65b3ac7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:57:12.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:12.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:12.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:57:12.611 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:57:12.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:12.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:14.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:14.064 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:57:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:57:14.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:14.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:14.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:15.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:15.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:15.489 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:57:15.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:57:15.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:15.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:16.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:17.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:17.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:17.078 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:57:17.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:57:17.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:17.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:17.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:18.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:18.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:18.620 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:57:18.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:57:18.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:18.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:19.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:20.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:20.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:20.271 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:57:20.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:57:20.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:20.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:20.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:21.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:21.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:21.692 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:57:21.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:57:21.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:21.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:22.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:23.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:23.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:23.236 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:57:23.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:57:23.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:23.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:23.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:24.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:24.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:24.778 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:57:24.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:57:24.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:24.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:25.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:26.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:26.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:26.687 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:57:26.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:57:26.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:26.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:27.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:28.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:28.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:28.382 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:57:28.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:57:28.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:28.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:29.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:30.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:30.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:30.025 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:57:30.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:57:30.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:30.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:30.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:31.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:31.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:31.833 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:57:31.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:57:31.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:31.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:32.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:33.489 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:57:33.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:57:33.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:33.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:34.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:35.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:35.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:35.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:57:35.404 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:57:35.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:35.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:57:35.823 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/57504914,v1:127.0.0.1:6811/57504914] [v2:127.0.0.1:6812/57504914,v1:127.0.0.1:6813/57504914] exists,up 8ff7c030-7d31-4d66-8456-36f599a7dac3 2024-09-16T15:57:35.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:57:35.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:57:35.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:57:35.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2024-09-16T15:57:35.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T15:57:35.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:57:35.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T15:57:35.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:57:35.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-16T15:57:35.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T15:57:35.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:57:35.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:57:35.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:57:35.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:57:35.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:57:35.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:57:35.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:57:35.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T15:57:35.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T15:57:35.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:57:35.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:57:35.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:57:35.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:57:35.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:57:35.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:57:35.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-16T15:57:35.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:57:35.829 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 5bc1523d-6273-4438-8e94-fba62146b971 2024-09-16T15:57:35.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5bc1523d-6273-4438-8e94-fba62146b971 2024-09-16T15:57:35.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5bc1523d-6273-4438-8e94-fba62146b971' 2024-09-16T15:57:35.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:57:35.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvVehmsu+kMhAAjz1gMQCA9pK7CrDntt+7mQ== 2024-09-16T15:57:35.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvVehmsu+kMhAAjz1gMQCA9pK7CrDntt+7mQ=="}' 2024-09-16T15:57:35.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5bc1523d-6273-4438-8e94-fba62146b971 -i td/osd-markdown/2/new.json 2024-09-16T15:57:36.364 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:57:36.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-16T15:57:36.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvVehmsu+kMhAAjz1gMQCA9pK7CrDntt+7mQ== --osd-uuid 5bc1523d-6273-4438-8e94-fba62146b971 2024-09-16T15:57:36.414 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:36.412+0000 7f444b7407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:36.416 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:36.412+0000 7f444b7407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:36.418 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:36.416+0000 7f444b7407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:36.418 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:36.416+0000 7f444b7407c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T15:57:36.418 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:36.416+0000 7f444b7407c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T15:57:36.418 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:36.416+0000 7f444b7407c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T15:57:36.479 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:36.476+0000 7f444b7407c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-16T15:57:42.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-16T15:57:42.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:57:42.370 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T15:57:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T15:57:42.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:57:42.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T15:57:42.933 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T15:57:42.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:57:42.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:57:42.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:57:42.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:57:42.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:42.960+0000 7fbd054507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:42.987 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:42.984+0000 7fbd054507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:42.989 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:42.988+0000 7fbd054507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:57:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T15:57:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:57:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T15:57:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:57:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:57:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:43.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:57:43.430 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:57:43.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:43.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:57:43.554 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:43.552+0000 7fbd054507c0 -1 Falling back to public interface 2024-09-16T15:57:43.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:44.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:44.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:44.848 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:57:44.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:57:44.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:44.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:57:45.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:46.065 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:57:46.060+0000 7fbd054507c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:57:46.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:46.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:46.296 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:57:46.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:57:46.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:46.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:57:46.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:47.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:47.762 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:57:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:57:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:47.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:57:48.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:49.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:49.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:49.195 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:57:49.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:57:49.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:49.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:57:49.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:50.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:50.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:50.728 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:57:50.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:57:50.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:50.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:57:51.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:52.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:52.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:52.192 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:57:52.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:57:52.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:52.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:57:52.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:57:53.631 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:57:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:57:54.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:55.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:55.222 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:57:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:57:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:57:55.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:56.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:56.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:56.674 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:57:56.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:57:56.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:56.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:57:57.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:58.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:58.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:58.153 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:57:58.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:57:58.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:58.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:57:58.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:57:59.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:57:59.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:57:59.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:57:59.574 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:57:59.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:57:59.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:58:00.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:58:01.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:58:01.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:58:01.587 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:58:01.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:58:01.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:58:01.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:58:02.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:58:03.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:58:03.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:58:03.578 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:58:03.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:58:03.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:58:03.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:58:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:58:05.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:58:05.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:58:05.681 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:58:05.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:58:05.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:58:05.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:58:06.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:58:07.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:58:07.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:58:07.427 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:58:07.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:58:07.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:58:07.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:58:08.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:58:09.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:58:09.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:58:09.445 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:58:09.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:58:09.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:58:09.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T15:58:09.903 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2763677642,v1:127.0.0.1:6819/2763677642] [v2:127.0.0.1:6820/2763677642,v1:127.0.0.1:6821/2763677642] exists,up 5bc1523d-6273-4438-8e94-fba62146b971 2024-09-16T15:58:09.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:58:09.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:58:09.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:58:09.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2024-09-16T15:58:09.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T15:58:10.315 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-09-16T15:58:10.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T15:58:10.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T15:58:10.896 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' already exists 2024-09-16T15:58:10.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T15:58:11.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T15:58:13.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2024-09-16T15:58:13.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2024-09-16T15:58:13.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2024-09-16T15:58:13.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-09-16T15:58:14.046 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T15:58:14.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2024-09-16T15:58:14.201 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T15:58:14.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2024-09-16T15:58:14.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-09-16T15:58:14.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2024-09-16T15:58:14.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-09-16T15:58:14.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-09-16T15:58:14.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-16T15:58:14.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-16T15:58:14.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-16T15:58:14.804 INFO:tasks.workunit.client.0.smithi039.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T15:58:14.805 INFO:tasks.workunit.client.0.smithi039.stdout:-1 0.29306 root default 2024-09-16T15:58:14.805 INFO:tasks.workunit.client.0.smithi039.stdout:-3 0.29306 host smithi039 2024-09-16T15:58:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:58:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-16T15:58:14.805 INFO:tasks.workunit.client.0.smithi039.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-16T15:58:14.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-16T15:58:14.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-16T15:58:14.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-16T15:58:15.298 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:58:15.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-16T15:58:15.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-16T15:58:15.724 INFO:tasks.workunit.client.0.smithi039.stderr:marked down osd.0. 2024-09-16T15:58:15.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-16T15:58:25.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-16T15:58:25.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-16T15:58:25.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-16T15:58:26.306 INFO:tasks.workunit.client.0.smithi039.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T15:58:26.306 INFO:tasks.workunit.client.0.smithi039.stdout:-1 0.29306 root default 2024-09-16T15:58:26.306 INFO:tasks.workunit.client.0.smithi039.stdout:-3 0.29306 host smithi039 2024-09-16T15:58:26.306 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:58:26.306 INFO:tasks.workunit.client.0.smithi039.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-16T15:58:26.306 INFO:tasks.workunit.client.0.smithi039.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-16T15:58:26.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-16T15:58:26.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-16T15:58:26.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-16T15:58:26.772 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:58:26.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-16T15:58:26.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-16T15:58:27.301 INFO:tasks.workunit.client.0.smithi039.stderr:marked down osd.0. 2024-09-16T15:58:27.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-16T15:58:37.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-16T15:58:37.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-16T15:58:37.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-16T15:58:37.936 INFO:tasks.workunit.client.0.smithi039.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T15:58:37.937 INFO:tasks.workunit.client.0.smithi039.stdout:-1 0.29306 root default 2024-09-16T15:58:37.937 INFO:tasks.workunit.client.0.smithi039.stdout:-3 0.29306 host smithi039 2024-09-16T15:58:37.937 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:58:37.937 INFO:tasks.workunit.client.0.smithi039.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-16T15:58:37.937 INFO:tasks.workunit.client.0.smithi039.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-16T15:58:37.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-16T15:58:37.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-16T15:58:37.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-16T15:58:38.377 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:58:38.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-16T15:58:38.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-16T15:58:38.813 INFO:tasks.workunit.client.0.smithi039.stderr:marked down osd.0. 2024-09-16T15:58:38.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-16T15:58:48.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-16T15:58:48.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-16T15:58:49.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-16T15:58:49.433 INFO:tasks.workunit.client.0.smithi039.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T15:58:49.433 INFO:tasks.workunit.client.0.smithi039.stdout:-1 0.29306 root default 2024-09-16T15:58:49.433 INFO:tasks.workunit.client.0.smithi039.stdout:-3 0.29306 host smithi039 2024-09-16T15:58:49.433 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:58:49.433 INFO:tasks.workunit.client.0.smithi039.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-16T15:58:49.433 INFO:tasks.workunit.client.0.smithi039.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-16T15:58:49.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-16T15:58:49.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-16T15:58:49.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-16T15:58:49.925 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:58:49.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-16T15:58:49.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-16T15:58:50.402 INFO:tasks.workunit.client.0.smithi039.stderr:marked down osd.0. 2024-09-16T15:58:50.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-16T15:59:00.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2024-09-16T15:59:15.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2024-09-16T15:59:15.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2024-09-16T15:59:15.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2024-09-16T15:59:15.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2024-09-16T15:59:15.983 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T15:59:15.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-16T15:59:15.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-16T15:59:15.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:59:15.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-16T15:59:15.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:59:15.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:59:15.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:59:15.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:59:15.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:59:16.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:59:16.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:59:16.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:59:16.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:59:16.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:59:16.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:59:16.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:59:16.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:59:16.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:59:16.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:59:16.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:59:16.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:59:16.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:59:16.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-16T15:59:16.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:59:16.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:59:16.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:59:16.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.177538 2024-09-16T15:59:16.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:59:16.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:59:16.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-16T15:59:16.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-16T15:59:16.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-16T15:59:16.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-16T15:59:16.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-16T15:59:16.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T15:59:16.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-16T15:59:16.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T15:59:16.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T15:59:16.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T15:59:16.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T15:59:16.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T15:59:16.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T15:59:16.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T15:59:16.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T15:59:16.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T15:59:16.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T15:59:16.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T15:59:16.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T15:59:16.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:59:16.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T15:59:16.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T15:59:16.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:59:16.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:59:16.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T15:59:16.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-16T15:59:16.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T15:59:16.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:59:16.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:59:16.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.177538 2024-09-16T15:59:16.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T15:59:16.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T15:59:16.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-16T15:59:16.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T15:59:16.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:59:16.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:59:16.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.177538 2024-09-16T15:59:16.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T15:59:16.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T15:59:16.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T15:59:16.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-16T15:59:16.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2024-09-16T15:59:16.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2024-09-16T15:59:16.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2024-09-16T15:59:16.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-16T15:59:16.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T15:59:16.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T15:59:16.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T15:59:16.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-16T15:59:16.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-16T15:59:16.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T15:59:16.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:59:16.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:59:16.739 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:59:16.740 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:59:16.740 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:59:16.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:59:16.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T15:59:16.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T15:59:16.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T15:59:16.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:59:16.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:59:16.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T15:59:16.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T15:59:16.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:59:16.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:59:16.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:59:16.946 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:59:16.946 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:59:16.946 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:59:16.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.177538/ceph-mon.a.asok 2024-09-16T15:59:16.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:59:16.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.177538/ceph-mon.a.asok config get fsid 2024-09-16T15:59:17.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T15:59:17.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T15:59:17.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T15:59:17.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T15:59:17.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T15:59:17.065 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T15:59:17.065 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T15:59:17.065 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T15:59:17.065 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T15:59:17.065 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:59:17.065 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:59:17.066 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.177538/ceph-mon.a.asok 2024-09-16T15:59:17.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T15:59:17.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.177538/ceph-mon.a.asok config get mon_host 2024-09-16T15:59:17.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2024-09-16T15:59:17.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-16T15:59:17.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T15:59:17.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T15:59:17.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T15:59:17.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-16T15:59:17.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T15:59:18.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T15:59:18.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:59:18.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:59:18.105 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:59:18.105 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:59:18.105 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:59:18.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:59:18.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:59:18.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T15:59:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2024-09-16T15:59:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T15:59:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:59:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:59:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:59:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:59:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:59:18.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:59:18.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:59:18.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:59:18.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:59:18.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:59:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T15:59:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T15:59:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:59:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:59:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:59:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:59:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:59:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:59:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-16T15:59:18.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:59:18.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=616f9358-b876-478c-80ca-15915b579d6b 2024-09-16T15:59:18.170 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 616f9358-b876-478c-80ca-15915b579d6b 2024-09-16T15:59:18.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 616f9358-b876-478c-80ca-15915b579d6b' 2024-09-16T15:59:18.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:59:18.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWVehmwbwTCxAA7oOhB6WBsecny3H2SQhW+Q== 2024-09-16T15:59:18.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWVehmwbwTCxAA7oOhB6WBsecny3H2SQhW+Q=="}' 2024-09-16T15:59:18.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 616f9358-b876-478c-80ca-15915b579d6b -i td/osd-markdown/0/new.json 2024-09-16T15:59:18.523 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:59:18.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-16T15:59:18.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWVehmwbwTCxAA7oOhB6WBsecny3H2SQhW+Q== --osd-uuid 616f9358-b876-478c-80ca-15915b579d6b 2024-09-16T15:59:18.572 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:18.570+0000 7f59af7187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:18.574 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:18.574+0000 7f59af7187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:18.576 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:18.574+0000 7f59af7187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:18.576 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:18.574+0000 7f59af7187c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T15:59:18.577 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:18.574+0000 7f59af7187c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T15:59:18.577 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:18.574+0000 7f59af7187c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T15:59:18.654 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:18.654+0000 7f59af7187c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-16T15:59:24.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-16T15:59:24.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:59:24.768 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T15:59:24.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T15:59:24.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:59:25.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T15:59:25.347 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T15:59:25.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:59:25.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:59:25.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:59:25.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:59:25.377 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:25.378+0000 7fac8ab857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:25.380 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:25.378+0000 7fac8ab857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:25.381 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:25.382+0000 7fac8ab857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:25.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T15:59:25.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:59:25.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T15:59:25.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:59:25.784 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:59:25.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:59:25.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:25.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:59:25.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:25.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:25.950 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:25.950+0000 7fac8ab857c0 -1 Falling back to public interface 2024-09-16T15:59:26.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:27.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:27.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:27.252 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:59:27.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T15:59:27.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:27.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:27.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:28.459 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:28.458+0000 7fac8ab857c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:59:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:28.726 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T15:59:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T15:59:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:28.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:29.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:30.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:30.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:30.298 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T15:59:30.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T15:59:30.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:30.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:30.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:31.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:31.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:31.725 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T15:59:31.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T15:59:31.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:31.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:32.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:33.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:33.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:33.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T15:59:33.240 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T15:59:33.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:33.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:33.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:34.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:34.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:34.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T15:59:34.679 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T15:59:34.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:34.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:35.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:36.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:36.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:36.186 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T15:59:36.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T15:59:36.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:36.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:36.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:37.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:37.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:37.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T15:59:37.610 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T15:59:37.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:37.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:38.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:39.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:39.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:39.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T15:59:39.186 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T15:59:39.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:39.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:39.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:40.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:40.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:40.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T15:59:40.676 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T15:59:40.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:40.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:41.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:42.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:42.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:42.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T15:59:42.142 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T15:59:42.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:42.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:42.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:43.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:43.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:43.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T15:59:43.585 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T15:59:43.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:43.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:44.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:45.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:45.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:45.505 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T15:59:45.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T15:59:45.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:45.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:46.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:47.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:47.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:47.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T15:59:47.351 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T15:59:47.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:47.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:48.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:49.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:49.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:49.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T15:59:49.649 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T15:59:49.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:49.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:50.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T15:59:51.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T15:59:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T15:59:51.103 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T15:59:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T15:59:51.584 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3445950392,v1:127.0.0.1:6803/3445950392] [v2:127.0.0.1:6804/3445950392,v1:127.0.0.1:6805/3445950392] exists,up 616f9358-b876-478c-80ca-15915b579d6b 2024-09-16T15:59:51.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T15:59:51.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T15:59:51.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T15:59:51.590 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 58354f60-9b89-49c6-bdad-f21d39d464c8 2024-09-16T15:59:51.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58354f60-9b89-49c6-bdad-f21d39d464c8 2024-09-16T15:59:51.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 58354f60-9b89-49c6-bdad-f21d39d464c8' 2024-09-16T15:59:51.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T15:59:51.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3VehmN4VVJBAA7iLx1f64Pap1GVlGjE+lPQ== 2024-09-16T15:59:51.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3VehmN4VVJBAA7iLx1f64Pap1GVlGjE+lPQ=="}' 2024-09-16T15:59:51.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58354f60-9b89-49c6-bdad-f21d39d464c8 -i td/osd-markdown/1/new.json 2024-09-16T15:59:52.141 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T15:59:52.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-16T15:59:52.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3VehmN4VVJBAA7iLx1f64Pap1GVlGjE+lPQ== --osd-uuid 58354f60-9b89-49c6-bdad-f21d39d464c8 2024-09-16T15:59:52.189 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:52.185+0000 7f9171dd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:52.191 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:52.189+0000 7f9171dd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:52.192 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:52.189+0000 7f9171dd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:52.193 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:52.189+0000 7f9171dd87c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T15:59:52.193 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:52.193+0000 7f9171dd87c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T15:59:52.193 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:52.193+0000 7f9171dd87c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T15:59:52.270 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:52.269+0000 7f9171dd87c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-16T15:59:58.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-16T15:59:58.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T15:59:58.639 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T15:59:58.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T15:59:58.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T15:59:59.212 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T15:59:59.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T15:59:59.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T15:59:59.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T15:59:59.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T15:59:59.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T15:59:59.243 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:59.241+0000 7f02693d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:59.245 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:59.245+0000 7f02693d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:59.246 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:59.245+0000 7f02693d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T15:59:59.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T15:59:59.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T15:59:59.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T15:59:59.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T15:59:59.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T15:59:59.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T15:59:59.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T15:59:59.694 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T15:59:59.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T15:59:59.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T15:59:59.794 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T15:59:59.793+0000 7f02693d37c0 -1 Falling back to public interface 2024-09-16T16:00:00.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:01.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:01.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:01.187 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:00:01.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:00:01.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:01.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:01.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:02.359 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:02.357+0000 7f02693d37c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:00:02.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:02.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:02.614 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:00:02.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:00:02.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:02.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:03.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:04.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:04.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:04.200 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:00:04.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:00:04.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:04.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:04.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:05.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:05.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:05.667 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:00:05.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:00:05.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:05.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:06.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:07.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:07.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:07.173 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:00:07.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:00:07.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:07.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:07.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:08.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:08.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:08.662 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:00:08.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:00:08.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:08.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:09.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:10.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:10.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:10.126 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:00:10.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:00:10.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:10.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:10.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:11.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:11.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:11.566 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:00:11.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:00:11.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:11.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:12.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:13.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:13.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:13.178 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:00:13.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:00:13.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:13.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:13.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:14.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:14.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:14.714 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:00:14.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:00:14.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:14.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:15.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:16.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:16.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:16.258 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:00:16.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:00:16.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:16.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:16.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:17.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:17.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:17.974 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:00:17.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:00:17.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:17.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:18.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:19.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:19.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:19.991 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:00:19.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:00:19.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:19.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:20.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:21.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:21.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:21.954 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:00:21.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:00:21.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:21.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:22.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:23.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:23.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:23.887 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:00:23.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:00:23.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:23.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:25.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:26.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:26.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:26.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:00:26.007 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:00:26.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:26.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:00:26.487 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3569408621,v1:127.0.0.1:6811/3569408621] [v2:127.0.0.1:6812/3569408621,v1:127.0.0.1:6813/3569408621] exists,up 58354f60-9b89-49c6-bdad-f21d39d464c8 2024-09-16T16:00:26.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:00:26.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:00:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-16T16:00:26.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-16T16:00:26.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:00:26.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:00:26.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T16:00:26.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:00:26.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:00:26.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:00:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:00:26.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:00:26.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:00:26.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:00:26.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-16T16:00:26.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:00:26.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=281c9cd0-908e-429b-bbe1-f8502114d7a3 2024-09-16T16:00:26.496 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 281c9cd0-908e-429b-bbe1-f8502114d7a3 2024-09-16T16:00:26.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 281c9cd0-908e-429b-bbe1-f8502114d7a3' 2024-09-16T16:00:26.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:00:26.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaVuhmg025HhAAVTQuahWVI1Tn3+VXhIMZeg== 2024-09-16T16:00:26.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaVuhmg025HhAAVTQuahWVI1Tn3+VXhIMZeg=="}' 2024-09-16T16:00:26.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 281c9cd0-908e-429b-bbe1-f8502114d7a3 -i td/osd-markdown/2/new.json 2024-09-16T16:00:27.042 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:00:27.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-16T16:00:27.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAaVuhmg025HhAAVTQuahWVI1Tn3+VXhIMZeg== --osd-uuid 281c9cd0-908e-429b-bbe1-f8502114d7a3 2024-09-16T16:00:27.090 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:27.088+0000 7fb7ae0a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:00:27.092 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:27.088+0000 7fb7ae0a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:00:27.094 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:27.092+0000 7fb7ae0a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:00:27.094 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:27.092+0000 7fb7ae0a27c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T16:00:27.094 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:27.092+0000 7fb7ae0a27c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T16:00:27.094 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:27.092+0000 7fb7ae0a27c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T16:00:27.188 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:27.184+0000 7fb7ae0a27c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-16T16:00:33.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-16T16:00:33.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:00:33.008 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T16:00:33.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T16:00:33.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:00:33.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T16:00:33.577 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T16:00:33.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:00:33.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:00:33.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:00:33.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:00:33.607 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:33.604+0000 7f2ae84ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:00:33.611 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:33.608+0000 7f2ae84ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:00:33.612 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:33.608+0000 7f2ae84ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:00:34.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T16:00:34.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:00:34.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T16:00:34.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:00:34.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:00:34.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:34.026 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:00:34.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:00:34.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:34.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:34.182 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:34.180+0000 7f2ae84ce7c0 -1 Falling back to public interface 2024-09-16T16:00:34.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:35.484 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:00:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:00:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:35.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:36.729 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:00:36.724+0000 7f2ae84ce7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T16:00:36.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:36.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:36.948 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:00:36.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:00:36.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:36.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:37.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:38.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:38.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:38.498 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:00:38.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:00:38.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:38.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:38.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:39.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:39.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:39.948 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:00:39.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:00:39.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:39.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:40.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:41.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:41.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:41.497 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:00:41.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:00:41.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:41.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:42.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:43.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:43.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:43.072 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:00:43.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:00:43.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:43.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:43.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:44.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:44.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:44.557 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:00:44.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:00:44.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:44.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:45.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:46.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:46.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:46.015 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:00:46.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:00:46.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:46.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:46.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:47.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:47.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:47.481 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:00:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:00:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:47.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:48.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:48.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:48.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:00:48.929 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:00:48.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:48.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:49.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:50.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:50.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:50.520 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:00:50.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:00:50.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:50.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:51.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:52.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:52.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:52.119 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:00:52.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:00:52.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:52.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:52.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:53.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:53.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:53.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:00:53.879 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:00:53.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:53.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:56.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:56.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:56.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:00:56.015 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:00:56.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:56.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:57.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:58.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:58.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:58.235 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:00:58.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:00:58.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:58.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:00:58.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:00:59.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:00:59.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:00:59.959 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:00:59.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:00:59.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:00:59.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:01:00.438 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1853606428,v1:127.0.0.1:6819/1853606428] [v2:127.0.0.1:6820/1853606428,v1:127.0.0.1:6821/1853606428] exists,up 281c9cd0-908e-429b-bbe1-f8502114d7a3 2024-09-16T16:01:00.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:01:00.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:01:00.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:01:00.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2024-09-16T16:01:00.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T16:01:00.915 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-09-16T16:01:00.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T16:01:00.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T16:01:01.812 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' already exists 2024-09-16T16:01:01.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:01:02.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T16:01:04.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2024-09-16T16:01:04.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2024-09-16T16:01:04.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2024-09-16T16:01:04.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-09-16T16:01:04.634 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T16:01:04.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2024-09-16T16:01:04.796 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T16:01:04.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2024-09-16T16:01:04.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-09-16T16:01:04.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2024-09-16T16:01:04.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-09-16T16:01:04.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-09-16T16:01:04.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-16T16:01:04.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-16T16:01:04.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-16T16:01:05.363 INFO:tasks.workunit.client.0.smithi039.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T16:01:05.363 INFO:tasks.workunit.client.0.smithi039.stdout:-1 0.29306 root default 2024-09-16T16:01:05.364 INFO:tasks.workunit.client.0.smithi039.stdout:-3 0.29306 host smithi039 2024-09-16T16:01:05.364 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T16:01:05.364 INFO:tasks.workunit.client.0.smithi039.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-16T16:01:05.364 INFO:tasks.workunit.client.0.smithi039.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-16T16:01:05.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-16T16:01:05.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-16T16:01:05.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-16T16:01:05.810 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T16:01:05.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-16T16:01:05.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-16T16:01:06.241 INFO:tasks.workunit.client.0.smithi039.stderr:marked down osd.0. 2024-09-16T16:01:06.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-16T16:01:16.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-16T16:01:16.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-16T16:01:16.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-16T16:01:16.860 INFO:tasks.workunit.client.0.smithi039.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T16:01:16.861 INFO:tasks.workunit.client.0.smithi039.stdout:-1 0.29306 root default 2024-09-16T16:01:16.861 INFO:tasks.workunit.client.0.smithi039.stdout:-3 0.29306 host smithi039 2024-09-16T16:01:16.861 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T16:01:16.861 INFO:tasks.workunit.client.0.smithi039.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-16T16:01:16.861 INFO:tasks.workunit.client.0.smithi039.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-16T16:01:16.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-16T16:01:16.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-16T16:01:16.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-16T16:01:17.328 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T16:01:17.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-16T16:01:17.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-16T16:01:17.780 INFO:tasks.workunit.client.0.smithi039.stderr:marked down osd.0. 2024-09-16T16:01:17.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-16T16:01:27.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-16T16:01:27.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-16T16:01:27.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-16T16:01:28.357 INFO:tasks.workunit.client.0.smithi039.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T16:01:28.358 INFO:tasks.workunit.client.0.smithi039.stdout:-1 0.29306 root default 2024-09-16T16:01:28.358 INFO:tasks.workunit.client.0.smithi039.stdout:-3 0.29306 host smithi039 2024-09-16T16:01:28.358 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T16:01:28.358 INFO:tasks.workunit.client.0.smithi039.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-16T16:01:28.358 INFO:tasks.workunit.client.0.smithi039.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-16T16:01:28.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-16T16:01:28.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-16T16:01:28.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-16T16:01:28.804 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T16:01:28.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-16T16:01:28.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-16T16:01:29.367 INFO:tasks.workunit.client.0.smithi039.stderr:marked down osd.0. 2024-09-16T16:01:29.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-16T16:01:39.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-16T16:01:39.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-16T16:01:39.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-16T16:01:39.996 INFO:tasks.workunit.client.0.smithi039.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T16:01:39.996 INFO:tasks.workunit.client.0.smithi039.stdout:-1 0.29306 root default 2024-09-16T16:01:39.996 INFO:tasks.workunit.client.0.smithi039.stdout:-3 0.29306 host smithi039 2024-09-16T16:01:39.996 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T16:01:39.996 INFO:tasks.workunit.client.0.smithi039.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-16T16:01:39.997 INFO:tasks.workunit.client.0.smithi039.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-16T16:01:40.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-16T16:01:40.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-16T16:01:40.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-16T16:01:40.432 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T16:01:40.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-16T16:01:40.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-16T16:01:40.857 INFO:tasks.workunit.client.0.smithi039.stderr:marked down osd.0. 2024-09-16T16:01:40.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-16T16:01:50.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2024-09-16T16:01:50.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2024-09-16T16:01:50.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2024-09-16T16:01:51.303 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-09-16T16:01:51.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-16T16:01:51.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-16T16:01:51.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:01:51.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-16T16:01:51.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:01:51.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:01:51.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:01:51.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:01:51.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:01:51.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:01:51.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:01:51.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:01:51.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:01:51.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:01:51.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:01:51.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:01:51.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:01:51.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:01:51.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:01:51.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:01:51.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:01:51.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:01:51.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-16T16:01:51.644 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:01:51.644 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:01:51.644 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:01:51.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.177538 2024-09-16T16:01:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:01:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:01:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-16T16:01:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-16T16:01:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-16T16:01:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-16T16:01:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-16T16:01:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:01:51.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-16T16:01:51.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:01:51.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:01:51.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:01:51.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:01:51.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:01:51.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:01:51.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:01:51.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:01:51.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:01:51.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:01:51.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:01:51.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:01:51.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:01:51.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:01:51.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:01:51.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:01:51.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:01:51.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:01:51.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-16T16:01:51.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:01:51.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:01:51.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:01:51.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.177538 2024-09-16T16:01:51.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:01:51.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:01:51.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-16T16:01:51.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:01:51.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:01:51.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:01:51.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.177538 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:01:51.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-16T16:01:51.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-16T16:01:52.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:01:52.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:01:52.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:01:52.040 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:01:52.040 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:01:52.041 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:01:52.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T16:01:52.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:01:52.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:01:52.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:01:52.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:01:52.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:01:52.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:01:52.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:01:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:01:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:01:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:01:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:01:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:01:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:01:52.230 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.177538/ceph-mon.a.asok 2024-09-16T16:01:52.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:01:52.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.177538/ceph-mon.a.asok config get fsid 2024-09-16T16:01:52.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:01:52.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:01:52.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:01:52.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:01:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:01:52.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:01:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:01:52.345 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:01:52.346 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:01:52.346 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:01:52.346 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:01:52.346 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.177538/ceph-mon.a.asok 2024-09-16T16:01:52.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:01:52.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.177538/ceph-mon.a.asok config get mon_host 2024-09-16T16:01:52.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2024-09-16T16:01:52.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-16T16:01:52.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:01:52.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:01:52.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:01:52.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-16T16:01:52.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:01:53.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:01:53.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:01:53.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:01:53.364 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:01:53.364 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:01:53.364 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:01:53.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T16:01:53.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:01:53.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:01:53.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2024-09-16T16:01:53.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T16:01:53.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:01:53.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:01:53.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:01:53.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:01:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:01:53.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:01:53.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-16T16:01:53.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:01:53.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d4d2c27c-cafd-4fef-a2c9-89883642e49c 2024-09-16T16:01:53.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d4d2c27c-cafd-4fef-a2c9-89883642e49c' 2024-09-16T16:01:53.406 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 d4d2c27c-cafd-4fef-a2c9-89883642e49c 2024-09-16T16:01:53.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:01:53.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxVuhmRUcWGRAAsDT3SaqrtBWOs1IrdkgM/A== 2024-09-16T16:01:53.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxVuhmRUcWGRAAsDT3SaqrtBWOs1IrdkgM/A=="}' 2024-09-16T16:01:53.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d4d2c27c-cafd-4fef-a2c9-89883642e49c -i td/osd-markdown/0/new.json 2024-09-16T16:01:53.782 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:01:53.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-16T16:01:53.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxVuhmRUcWGRAAsDT3SaqrtBWOs1IrdkgM/A== --osd-uuid d4d2c27c-cafd-4fef-a2c9-89883642e49c 2024-09-16T16:01:53.836 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:01:53.834+0000 7fd4637877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:01:53.856 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:01:53.854+0000 7fd4637877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:01:53.858 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:01:53.858+0000 7fd4637877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:01:53.858 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:01:53.858+0000 7fd4637877c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T16:01:53.858 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:01:53.858+0000 7fd4637877c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T16:01:53.859 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:01:53.858+0000 7fd4637877c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-16T16:01:53.939 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:01:53.938+0000 7fd4637877c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-16T16:01:59.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-16T16:01:59.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:01:59.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:01:59.866 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:01:59.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:02:00.433 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:02:00.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:02:00.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:02:00.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:02:00.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:02:00.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:02:00.463 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:00.462+0000 7f14cb1b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:02:00.481 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:00.478+0000 7f14cb1b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:02:00.482 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:00.482+0000 7f14cb1b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:02:00.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:02:00.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:02:00.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:02:00.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:02:00.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:02:00.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:00.855 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:02:00.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:02:00.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:00.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:01.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:01.038+0000 7f14cb1b07c0 -1 Falling back to public interface 2024-09-16T16:02:01.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:02.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:02.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:02.263 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:02:02.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:02:02.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:02.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:02.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:03.573 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:03.570+0000 7f14cb1b07c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:02:03.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:03.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:03.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:02:03.725 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:02:03.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:03.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:04.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:05.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:05.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:05.228 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:02:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:02:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:05.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:06.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:06.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:06.688 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:02:06.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:02:06.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:06.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:07.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:08.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:08.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:08.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:02:08.109 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:02:08.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:08.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:08.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:09.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:09.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:09.572 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:02:09.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:02:09.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:09.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:10.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:11.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:11.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:11.040 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:02:11.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:02:11.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:11.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:11.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:12.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:12.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:12.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:02:12.611 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:02:12.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:12.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:02:14.065 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:02:14.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:14.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:14.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:15.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:15.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:15.581 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:02:15.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:02:15.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:15.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:16.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:17.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:17.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:17.369 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:02:17.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:02:17.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:17.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:18.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:19.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:19.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:19.314 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:02:19.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:02:19.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:19.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:20.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:21.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:21.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:21.195 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:02:21.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:02:21.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:21.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:21.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:22.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:22.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:22.918 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:02:22.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:02:22.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:22.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:23.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:24.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:24.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:24.687 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:02:24.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:02:24.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:24.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:25.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:26.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:26.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:26.539 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:02:26.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:02:26.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:26.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:02:26.967 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2735392094,v1:127.0.0.1:6803/2735392094] [v2:127.0.0.1:6804/2735392094,v1:127.0.0.1:6805/2735392094] exists,up d4d2c27c-cafd-4fef-a2c9-89883642e49c 2024-09-16T16:02:26.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:02:26.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:02:26.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:02:26.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2024-09-16T16:02:26.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T16:02:26.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:02:26.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:02:26.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:02:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:02:26.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:02:26.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:02:26.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:02:26.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-16T16:02:26.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:02:26.972 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 aa768322-991e-46c4-b7d8-99cabaa5f7d1 2024-09-16T16:02:26.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa768322-991e-46c4-b7d8-99cabaa5f7d1 2024-09-16T16:02:26.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 aa768322-991e-46c4-b7d8-99cabaa5f7d1' 2024-09-16T16:02:26.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:02:26.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSVuhmToXUOhAAJPhcq3QdCFyEmDK0DhpH7A== 2024-09-16T16:02:26.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSVuhmToXUOhAAJPhcq3QdCFyEmDK0DhpH7A=="}' 2024-09-16T16:02:26.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa768322-991e-46c4-b7d8-99cabaa5f7d1 -i td/osd-markdown/1/new.json 2024-09-16T16:02:27.544 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:02:27.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-16T16:02:27.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSVuhmToXUOhAAJPhcq3QdCFyEmDK0DhpH7A== --osd-uuid aa768322-991e-46c4-b7d8-99cabaa5f7d1 2024-09-16T16:02:27.594 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:27.594+0000 7ff1fdf8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:02:27.596 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:27.594+0000 7ff1fdf8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:02:27.597 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:27.594+0000 7ff1fdf8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:02:27.598 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:27.598+0000 7ff1fdf8c7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T16:02:27.598 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:27.598+0000 7ff1fdf8c7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T16:02:27.598 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:27.598+0000 7ff1fdf8c7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-16T16:02:27.675 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:27.674+0000 7ff1fdf8c7c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-16T16:02:33.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-16T16:02:33.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:02:33.821 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:02:33.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:02:33.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:02:34.402 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:02:34.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:02:34.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:02:34.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:02:34.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:02:34.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:02:34.432 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:34.429+0000 7f66023837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:02:34.435 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:34.433+0000 7f66023837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:02:34.436 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:34.433+0000 7f66023837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:02:34.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:02:34.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:02:34.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:02:34.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:02:34.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:02:34.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:34.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:02:34.881 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:02:34.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:34.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:34.980 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:34.977+0000 7f66023837c0 -1 Falling back to public interface 2024-09-16T16:02:35.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:36.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:36.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:36.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:02:36.364 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:02:36.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:36.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:36.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:37.469 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:02:37.465+0000 7f66023837c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:02:37.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:37.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:37.834 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:02:37.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:02:37.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:37.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:38.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:39.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:39.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:39.265 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:02:39.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:02:39.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:39.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:39.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:40.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:40.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:40.719 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:02:40.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:02:40.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:40.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:41.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:42.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:42.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:42.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:02:42.256 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:02:42.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:42.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:42.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:43.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:43.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:43.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:02:43.825 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:02:43.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:43.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:44.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:45.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:45.373 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:02:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:02:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:45.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:45.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:46.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:46.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:46.840 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:02:46.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:02:46.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:46.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:47.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:48.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:48.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:48.374 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:02:48.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:02:48.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:48.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:48.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:49.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:49.842 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:02:49.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:49.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:02:49.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:49.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:50.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:51.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:51.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:51.333 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:02:51.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:02:51.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:51.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:52.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:53.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:53.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:53.134 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:02:53.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:02:53.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:53.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:53.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:54.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:54.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:54.883 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:02:54.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:02:54.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:54.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:55.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:56.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:56.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:56.630 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:02:56.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:02:56.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:56.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:02:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:02:59.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:02:59.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:02:59.017 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:02:59.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:02:59.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:02:59.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:03:00.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:01.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:01.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:01.105 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:03:01.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:03:01.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:01.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:03:01.535 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1974961712,v1:127.0.0.1:6811/1974961712] [v2:127.0.0.1:6812/1974961712,v1:127.0.0.1:6813/1974961712] exists,up aa768322-991e-46c4-b7d8-99cabaa5f7d1 2024-09-16T16:03:01.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:03:01.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:03:01.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:03:01.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-16T16:03:01.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:03:01.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:03:01.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-16T16:03:01.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:03:01.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:03:01.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:03:01.539 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:03:01.539 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:03:01.539 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:03:01.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T16:03:01.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' 2024-09-16T16:03:01.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:03:01.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:03:01.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:03:01.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-16T16:03:01.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-16T16:03:01.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:03:01.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:03:01.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:03:01.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:03:01.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:03:01.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:03:01.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-16T16:03:01.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:03:01.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0d6bea8-6070-47d8-9e4a-5074f822d188 2024-09-16T16:03:01.542 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 b0d6bea8-6070-47d8-9e4a-5074f822d188 2024-09-16T16:03:01.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b0d6bea8-6070-47d8-9e4a-5074f822d188' 2024-09-16T16:03:01.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:03:01.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1VuhmjCCIIRAAI/wkA/dAsG8/preuSxeKDg== 2024-09-16T16:03:01.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1VuhmjCCIIRAAI/wkA/dAsG8/preuSxeKDg=="}' 2024-09-16T16:03:01.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0d6bea8-6070-47d8-9e4a-5074f822d188 -i td/osd-markdown/2/new.json 2024-09-16T16:03:02.136 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:03:02.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-16T16:03:02.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1VuhmjCCIIRAAI/wkA/dAsG8/preuSxeKDg== --osd-uuid b0d6bea8-6070-47d8-9e4a-5074f822d188 2024-09-16T16:03:02.192 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:02.189+0000 7f6d0c3cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:03:02.217 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:02.213+0000 7f6d0c3cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:03:02.219 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:02.217+0000 7f6d0c3cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:03:02.219 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:02.217+0000 7f6d0c3cc7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T16:03:02.219 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:02.217+0000 7f6d0c3cc7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T16:03:02.219 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:02.217+0000 7f6d0c3cc7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-16T16:03:02.300 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:02.297+0000 7f6d0c3cc7c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-16T16:03:08.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-16T16:03:08.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:03:08.363 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T16:03:08.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T16:03:08.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:03:08.942 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T16:03:08.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T16:03:08.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=452c3d98-bd09-4b22-97e4-812d67e4d7a2 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.177538/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:03:08.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:03:08.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:03:08.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:03:08.973 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:08.969+0000 7f60602aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:03:09.000 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:08.997+0000 7f60602aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:03:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:08.997+0000 7f60602aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:03:09.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T16:03:09.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:03:09.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T16:03:09.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:03:09.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:03:09.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:09.377 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:03:09.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:03:09.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:09.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:09.579 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:09.577+0000 7f60602aa7c0 -1 Falling back to public interface 2024-09-16T16:03:09.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:10.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:10.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:10.792 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:03:10.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:03:10.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:10.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:11.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:12.136 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:12.133+0000 7f60602aa7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T16:03:12.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:12.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:12.261 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:03:12.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:03:12.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:12.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:12.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:13.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:13.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:13.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:03:13.833 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:03:13.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:13.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:14.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:15.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:15.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:15.288 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:03:15.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:03:15.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:15.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:15.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:16.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:16.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:16.741 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:03:16.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:03:16.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:16.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:17.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:18.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:18.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:18.184 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:03:18.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:03:18.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:18.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:18.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:19.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:19.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:19.730 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:03:19.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:03:19.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:19.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:20.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:21.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:21.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:21.298 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:03:21.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:03:21.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:21.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:21.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:22.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:22.718 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:03:22.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:22.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:03:22.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:22.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:23.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:24.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:24.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:24.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:03:24.192 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:03:24.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:24.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:25.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:26.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:26.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:26.136 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:03:26.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:03:26.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:26.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:26.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:27.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:27.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:27.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:03:27.943 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:03:27.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:27.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:28.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:29.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:29.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:29.595 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:03:29.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:03:29.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:29.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:30.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:31.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:31.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:31.348 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:03:31.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:03:31.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:31.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:32.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:33.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:33.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:33.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:03:33.224 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:03:33.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:33.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:34.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:03:35.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:03:35.174 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:03:35.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:03:35.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:03:35.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:03:35.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:03:35.620 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1464006206,v1:127.0.0.1:6819/1464006206] [v2:127.0.0.1:6820/1464006206,v1:127.0.0.1:6821/1464006206] exists,up b0d6bea8-6070-47d8-9e4a-5074f822d188 2024-09-16T16:03:35.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:03:35.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:03:35.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:03:35.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2024-09-16T16:03:35.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=196371 2024-09-16T16:03:35.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 196371 2024-09-16T16:03:35.630 INFO:tasks.workunit.client.0.smithi039.stdout: PID TTY TIME CMD 2024-09-16T16:03:35.630 INFO:tasks.workunit.client.0.smithi039.stdout: 196371 ? 00:00:00 ceph-osd 2024-09-16T16:03:35.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2024-09-16T16:03:35.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2024-09-16T16:03:35.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2024-09-16T16:03:36.051 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-16T16:03:36.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2024-09-16T16:03:36.759 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already stopped. 2024-09-16T16:03:36.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2024-09-16T16:03:51.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2024-09-16T16:03:51.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2024-09-16T16:03:51.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2024-09-16T16:03:52.252 INFO:tasks.workunit.client.0.smithi039.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-09-16T16:03:52.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 196371 2024-09-16T16:03:52.262 INFO:tasks.workunit.client.0.smithi039.stdout: PID TTY TIME CMD 2024-09-16T16:03:52.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-16T16:03:52.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-16T16:03:52.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:03:52.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-16T16:03:52.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:03:52.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:03:52.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:03:52.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:03:52.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:03:52.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:03:52.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:03:52.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:03:52.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:03:52.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:03:52.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:03:52.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:03:52.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:03:52.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:03:52.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:03:52.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:03:52.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:03:52.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:03:52.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-16T16:03:52.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:03:52.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:03:52.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:03:52.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.177538 2024-09-16T16:03:52.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:03:52.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:03:52.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T16:03:52.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-markdown 0 2024-09-16T16:03:52.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-16T16:03:52.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T16:03:52.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-16T16:03:52.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:03:52.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:03:52.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:03:52.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:03:52.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:03:52.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:03:52.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:03:52.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:03:52.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:03:52.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:03:52.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:03:52.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:03:52.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:03:52.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:03:52.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:03:52.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:03:52.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:03:52.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T16:03:52.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-16T16:03:52.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:03:52.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:03:52.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.177538 2024-09-16T16:03:52.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.177538 2024-09-16T16:03:52.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:03:52.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:03:52.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T16:03:52.620 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T16:03:52.621 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T16:03:52.674 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2024-09-16T16:03:52.674 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2024-09-16T16:03:52.730 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-reactivate 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2024-09-16T16:03:52.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2024-09-16T16:03:52.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2024-09-16T16:03:52.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2024-09-16T16:03:52.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2024-09-16T16:03:52.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=f5cb93c2-e809-4a44-a4e7-a39b03666e1d --auth-supported=none ' 2024-09-16T16:03:52.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2024-09-16T16:03:52.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2024-09-16T16:03:52.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T16:03:52.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2024-09-16T16:03:52.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2024-09-16T16:03:52.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2024-09-16T16:03:52.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2024-09-16T16:03:52.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2024-09-16T16:03:52.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-09-16T16:03:52.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:03:52.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-09-16T16:03:52.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:03:52.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:03:52.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:03:52.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:03:52.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:03:52.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:03:52.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:03:52.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:03:52.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:03:52.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:03:52.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:03:52.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:03:52.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:03:52.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:03:52.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:03:52.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:03:52.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:03:52.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:03:52.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-09-16T16:03:52.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:03:52.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:03:52.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200811 2024-09-16T16:03:52.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.200811 2024-09-16T16:03:52.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:03:52.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:03:52.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2024-09-16T16:03:52.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:03:52.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:03:52.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200811 2024-09-16T16:03:52.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.200811 2024-09-16T16:03:52.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:03:52.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T16:03:52.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T16:03:52.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:03:52.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2024-09-16T16:03:52.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2024-09-16T16:03:52.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2024-09-16T16:03:52.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2024-09-16T16:03:52.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2024-09-16T16:03:52.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:03:52.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:03:52.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:03:52.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2024-09-16T16:03:52.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2024-09-16T16:03:53.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:03:53.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:03:53.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:03:53.149 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:03:53.149 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:03:53.149 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200811 2024-09-16T16:03:53.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.200811/$cluster-$name.asok' 2024-09-16T16:03:53.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.200811/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:03:53.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:03:53.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:03:53.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:03:53.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:03:53.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:03:53.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:03:53.354 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:03:53.354 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:03:53.354 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:03:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:03:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:03:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200811 2024-09-16T16:03:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.200811/ceph-mon.a.asok 2024-09-16T16:03:53.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:03:53.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.200811/ceph-mon.a.asok config get fsid 2024-09-16T16:03:53.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:03:53.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:03:53.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:03:53.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:03:53.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:03:53.474 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:03:53.474 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:03:53.475 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:03:53.475 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:03:53.475 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:03:53.475 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200811 2024-09-16T16:03:53.475 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.200811/ceph-mon.a.asok 2024-09-16T16:03:53.475 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:03:53.475 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.200811/ceph-mon.a.asok config get mon_host 2024-09-16T16:03:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2024-09-16T16:03:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2024-09-16T16:03:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:03:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:03:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:03:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2024-09-16T16:03:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:03:54.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:03:54.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:03:54.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:03:54.511 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:03:54.511 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:03:54.511 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200811 2024-09-16T16:03:54.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.200811/$cluster-$name.asok' 2024-09-16T16:03:54.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:03:54.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.200811/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:03:54.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2024-09-16T16:03:54.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2024-09-16T16:03:54.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:03:54.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:03:54.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:03:54.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2024-09-16T16:03:54.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f5cb93c2-e809-4a44-a4e7-a39b03666e1d --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-09-16T16:03:54.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:03:54.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:03:54.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:03:54.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-09-16T16:03:54.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-09-16T16:03:54.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:03:54.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:03:54.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-09-16T16:03:54.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:03:54.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:03:54.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:03:54.579 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:03:54.579 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:03:54.579 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200811 2024-09-16T16:03:54.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.200811/$cluster-$name.asok' 2024-09-16T16:03:54.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.200811/$cluster-$name.asok' 2024-09-16T16:03:54.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:03:54.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:03:54.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:03:54.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-09-16T16:03:54.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-09-16T16:03:54.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:03:54.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:03:54.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:03:54.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:03:54.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:03:54.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:03:54.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reactivate/0 2024-09-16T16:03:54.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:03:54.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1ddaacc-5217-42ef-bf96-665eea62e548 2024-09-16T16:03:54.581 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 d1ddaacc-5217-42ef-bf96-665eea62e548 2024-09-16T16:03:54.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d1ddaacc-5217-42ef-bf96-665eea62e548' 2024-09-16T16:03:54.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:03:54.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDqVuhmgt51IxAAJ8z9UQsQbpoC6gaGJpaM1w== 2024-09-16T16:03:54.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDqVuhmgt51IxAAJ8z9UQsQbpoC6gaGJpaM1w=="}' 2024-09-16T16:03:54.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1ddaacc-5217-42ef-bf96-665eea62e548 -i td/osd-reactivate/0/new.json 2024-09-16T16:03:54.906 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:03:54.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reactivate/0/new.json 2024-09-16T16:03:54.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f5cb93c2-e809-4a44-a4e7-a39b03666e1d --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.200811/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqVuhmgt51IxAAJ8z9UQsQbpoC6gaGJpaM1w== --osd-uuid d1ddaacc-5217-42ef-bf96-665eea62e548 2024-09-16T16:03:54.957 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:54.952+0000 7f997e4287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:03:54.958 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:54.956+0000 7f997e4287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:03:54.960 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:54.956+0000 7f997e4287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:03:54.960 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:54.956+0000 7f997e4287c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-09-16T16:03:54.960 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:54.956+0000 7f997e4287c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-09-16T16:03:54.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:54.956+0000 7f997e4287c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-09-16T16:03:54.997 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:03:54.992+0000 7f997e4287c0 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2024-09-16T16:04:01.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reactivate/0/keyring 2024-09-16T16:04:01.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:04:01.184 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:04:01.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:04:01.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:04:01.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:04:01.743 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:04:01.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f5cb93c2-e809-4a44-a4e7-a39b03666e1d --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.200811/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:04:01.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:04:01.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:04:01.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:04:01.773 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:01.767+0000 7f107283e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:01.779 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:01.775+0000 7f107283e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:01.781 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:01.775+0000 7f107283e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:02.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:04:02.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:04:02.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:04:02.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:04:02.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:04:02.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:02.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:04:02.163 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:04:02.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:02.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:02.364 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:02.359+0000 7f107283e7c0 -1 Falling back to public interface 2024-09-16T16:04:02.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:03.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:03.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:03.610 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:04:03.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:04:03.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:03.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:04.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:04.965 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:04.959+0000 7f107283e7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:04:05.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:05.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:05.073 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:04:05.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:04:05.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:05.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:05.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:06.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:06.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:06.650 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:04:06.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:04:06.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:06.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:07.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:08.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:08.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:08.080 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:04:08.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:04:08.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:08.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:08.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:09.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:09.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:09.554 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:04:09.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:04:09.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:09.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:10.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:11.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:11.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:11.027 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:04:11.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:04:11.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:11.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:11.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:12.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:12.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:12.459 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:04:12.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:04:12.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:12.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:13.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:14.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:14.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:14.044 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:04:14.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:04:14.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:14.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:14.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:15.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:15.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:15.475 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:04:15.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:04:15.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:15.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:16.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:17.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:17.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:17.031 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:04:17.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:04:17.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:17.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:17.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:18.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:18.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:18.527 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:04:18.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:04:18.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:18.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:19.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:20.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:20.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:20.020 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:04:20.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:04:20.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:20.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:20.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:21.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:21.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:21.688 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:04:21.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:04:21.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:21.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:22.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:23.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:23.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:23.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:04:23.596 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:04:23.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:23.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:24.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:25.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:25.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:25.659 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:04:25.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:04:25.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:25.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:26.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:27.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:27.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:27.907 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:04:27.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:04:27.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:27.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:28.375 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3363614756,v1:127.0.0.1:6803/3363614756] [v2:127.0.0.1:6804/3363614756,v1:127.0.0.1:6805/3363614756] exists,up d1ddaacc-5217-42ef-bf96-665eea62e548 2024-09-16T16:04:28.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:04:28.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:04:28.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:04:28.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2024-09-16T16:04:28.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:04:28.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:04:28.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:04:28.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:04:28.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:04:28.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:04:28.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2024-09-16T16:04:28.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2024-09-16T16:04:28.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2024-09-16T16:04:28.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2024-09-16T16:04:28.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-reactivate 2024-09-16T16:04:28.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T16:04:28.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T16:04:28.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T16:04:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-reactivate/0 2024-09-16T16:04:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f5cb93c2-e809-4a44-a4e7-a39b03666e1d --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-09-16T16:04:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:04:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:04:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:04:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-09-16T16:04:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-09-16T16:04:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T16:04:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T16:04:28.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-09-16T16:04:28.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T16:04:28.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:04:28.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:04:28.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:04:28.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:04:28.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200811 2024-09-16T16:04:28.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.200811/$cluster-$name.asok' 2024-09-16T16:04:28.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.200811/$cluster-$name.asok' 2024-09-16T16:04:28.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:04:28.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-09-16T16:04:28.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-09-16T16:04:28.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:04:28.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:04:28.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:04:28.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:04:28.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T16:04:28.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T16:04:28.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-reactivate/0 2024-09-16T16:04:28.696 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:04:28.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T16:04:28.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f5cb93c2-e809-4a44-a4e7-a39b03666e1d --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.200811/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:04:28.698 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-reactivate/0/whoami 2024-09-16T16:04:28.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T16:04:28.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T16:04:28.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T16:04:28.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T16:04:28.727 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:28.723+0000 7f472c3267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:28.756 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:28.751+0000 7f472c3267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:28.758 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:28.751+0000 7f472c3267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:29.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T16:04:29.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:04:29.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:04:29.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:04:29.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:04:29.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:29.170 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:04:29.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:04:29.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:29.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:29.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:29.323+0000 7f472c3267c0 -1 Falling back to public interface 2024-09-16T16:04:29.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:30.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:30.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:30.654 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:04:30.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:04:30.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:30.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:31.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:32.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:32.080 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:04:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:04:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:32.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:32.305 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:32.303+0000 7f472c3267c0 -1 osd.0 6 log_to_monitors true 2024-09-16T16:04:32.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:33.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:33.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:33.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:04:33.512 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:04:33.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:33.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:33.995 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/1674344910,v1:127.0.0.1:6803/1674344910] [v2:127.0.0.1:6804/1674344910,v1:127.0.0.1:6805/1674344910] exists,up d1ddaacc-5217-42ef-bf96-665eea62e548 2024-09-16T16:04:33.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:04:33.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:04:33.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:04:33.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2024-09-16T16:04:33.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-09-16T16:04:33.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:04:33.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-09-16T16:04:33.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:04:33.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:04:33.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:04:33.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:04:33.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:04:34.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:04:34.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:04:34.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:04:34.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:04:34.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:04:34.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:04:34.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:04:34.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:04:34.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:04:34.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:04:34.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:04:34.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:04:34.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:04:34.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-09-16T16:04:34.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:04:34.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:04:34.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200811 2024-09-16T16:04:34.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.200811 2024-09-16T16:04:34.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:04:34.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:04:34.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T16:04:34.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-reactivate 0 2024-09-16T16:04:34.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-09-16T16:04:34.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T16:04:34.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-09-16T16:04:34.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:04:34.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:04:34.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:04:34.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:04:34.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:04:34.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:04:34.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:04:34.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:04:34.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:04:34.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:04:34.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:04:34.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:04:34.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:04:34.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:04:34.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:04:34.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:04:34.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:04:34.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T16:04:34.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-09-16T16:04:34.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:04:34.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:04:34.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200811 2024-09-16T16:04:34.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.200811 2024-09-16T16:04:34.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:04:34.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:04:34.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T16:04:34.337 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T16:04:34.338 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T16:04:34.393 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2024-09-16T16:04:34.394 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2024-09-16T16:04:34.453 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T16:04:34.460 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T16:04:34.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T16:04:34.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T16:04:34.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T16:04:34.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T16:04:34.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T16:04:34.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T16:04:34.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T16:04:34.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T16:04:34.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-recovery-prio 2024-09-16T16:04:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2024-09-16T16:04:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2024-09-16T16:04:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2024-09-16T16:04:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2024-09-16T16:04:34.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2024-09-16T16:04:34.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2024-09-16T16:04:34.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none ' 2024-09-16T16:04:34.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2024-09-16T16:04:34.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-09-16T16:04:34.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export objects=200 2024-09-16T16:04:34.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: objects=200 2024-09-16T16:04:34.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: export poolprefix=test 2024-09-16T16:04:34.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: poolprefix=test 2024-09-16T16:04:34.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: export FORCE_PRIO=255 2024-09-16T16:04:34.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: FORCE_PRIO=255 2024-09-16T16:04:34.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: export NORMAL_PRIO=190 2024-09-16T16:04:34.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: NORMAL_PRIO=190 2024-09-16T16:04:34.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: set 2024-09-16T16:04:34.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T16:04:34.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: local 'funcs=TEST_recovery_pool_priority 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_recovery_priority' 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:04:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:04:34.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:04:34.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:04:34.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:04:34.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:04:34.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:04:34.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:04:34.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:04:34.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:04:34.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:04:34.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:04:34.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:04:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:04:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:04:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-16T16:04:34.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:04:34.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:04:34.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:04:34.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.203397 2024-09-16T16:04:34.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:04:34.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:04:34.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-09-16T16:04:34.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:04:34.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:04:34.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:04:34.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.203397 2024-09-16T16:04:34.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:04:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T16:04:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T16:04:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:04:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-09-16T16:04:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_pool_priority td/osd-recovery-prio 2024-09-16T16:04:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:326: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2024-09-16T16:04:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: local pools=3 2024-09-16T16:04:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: local OSDS=2 2024-09-16T16:04:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:330: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2024-09-16T16:04:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-09-16T16:04:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:04:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:04:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:04:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-09-16T16:04:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-09-16T16:04:34.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:04:34.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:04:34.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:04:34.879 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:04:34.879 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:04:34.879 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:04:34.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:04:34.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:04:35.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:04:35.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:04:35.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:04:35.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:04:35.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:04:35.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:04:35.092 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:04:35.092 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:04:35.092 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:04:35.092 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:04:35.093 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:04:35.093 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:04:35.093 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-mon.a.asok 2024-09-16T16:04:35.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:04:35.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.203397/ceph-mon.a.asok config get fsid 2024-09-16T16:04:35.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:04:35.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-mon.a.asok 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:04:35.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.203397/ceph-mon.a.asok config get mon_host 2024-09-16T16:04:35.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2024-09-16T16:04:35.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-09-16T16:04:35.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:04:35.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:04:35.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:04:35.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-09-16T16:04:35.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:04:36.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:04:36.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:04:36.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:04:36.221 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:04:36.222 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:04:36.222 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:04:36.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:04:36.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:04:36.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:04:36.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:332: TEST_recovery_pool_priority: export CEPH_ARGS 2024-09-16T16:04:36.257 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: expr 2 - 1 2024-09-16T16:04:36.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: seq 0 1 2024-09-16T16:04:36.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T16:04:36.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2024-09-16T16:04:36.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-16T16:04:36.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:04:36.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:04:36.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:04:36.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-09-16T16:04:36.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:04:36.262 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-09-16T16:04:36.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:04:36.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=41670c1f-c16a-432d-bfa2-6b4779ce58c8 2024-09-16T16:04:36.264 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 41670c1f-c16a-432d-bfa2-6b4779ce58c8 2024-09-16T16:04:36.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 41670c1f-c16a-432d-bfa2-6b4779ce58c8' 2024-09-16T16:04:36.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:04:36.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUV+hm9bPGEBAALITb59YujrmsGMryDKUfYA== 2024-09-16T16:04:36.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUV+hm9bPGEBAALITb59YujrmsGMryDKUfYA=="}' 2024-09-16T16:04:36.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 41670c1f-c16a-432d-bfa2-6b4779ce58c8 -i td/osd-recovery-prio/0/new.json 2024-09-16T16:04:36.637 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:04:36.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-09-16T16:04:36.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUV+hm9bPGEBAALITb59YujrmsGMryDKUfYA== --osd-uuid 41670c1f-c16a-432d-bfa2-6b4779ce58c8 2024-09-16T16:04:36.688 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:36.687+0000 7feef06357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:36.690 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:36.691+0000 7feef06357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:36.692 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:36.691+0000 7feef06357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:36.692 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:36.691+0000 7feef06357c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-16T16:04:36.692 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:36.691+0000 7feef06357c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-16T16:04:36.692 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:36.691+0000 7feef06357c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-16T16:04:36.769 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:36.767+0000 7feef06357c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-09-16T16:04:42.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-09-16T16:04:42.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:04:42.950 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:04:42.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:04:42.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:04:43.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:04:43.510 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:04:43.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:04:43.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:04:43.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:04:43.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:04:43.545 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:43.542+0000 7f830dc117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:43.558 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:43.558+0000 7f830dc117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:43.560 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:43.558+0000 7f830dc117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:43.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:04:43.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:04:43.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:04:43.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:04:43.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:04:43.954 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:04:43.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:43.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:04:43.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:43.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:44.127 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:44.126+0000 7f830dc117c0 -1 Falling back to public interface 2024-09-16T16:04:44.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:45.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:45.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:45.396 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:04:45.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:04:45.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:45.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:45.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:46.667 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:46.666+0000 7f830dc117c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:04:46.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:46.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:46.885 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:04:46.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:04:46.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:46.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:04:47.477 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1330223197,v1:127.0.0.1:6803/1330223197] [v2:127.0.0.1:6804/1330223197,v1:127.0.0.1:6805/1330223197] exists,up 41670c1f-c16a-432d-bfa2-6b4779ce58c8 2024-09-16T16:04:47.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:04:47.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:04:47.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:04:47.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T16:04:47.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2024-09-16T16:04:47.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-16T16:04:47.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:04:47.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:04:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:04:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-09-16T16:04:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-16T16:04:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:04:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:04:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:04:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-09-16T16:04:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-09-16T16:04:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:04:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:04:47.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:04:47.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-16T16:04:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-16T16:04:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:04:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:04:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:04:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:04:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:04:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:04:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-09-16T16:04:47.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:04:47.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b56fc19a-5b8d-4522-981f-d88ff8769ae9 2024-09-16T16:04:47.485 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 b56fc19a-5b8d-4522-981f-d88ff8769ae9 2024-09-16T16:04:47.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b56fc19a-5b8d-4522-981f-d88ff8769ae9' 2024-09-16T16:04:47.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:04:47.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfV+hmZRE6HhAA0k3uRK/+J2Y0UMFTqqQmAQ== 2024-09-16T16:04:47.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfV+hmZRE6HhAA0k3uRK/+J2Y0UMFTqqQmAQ=="}' 2024-09-16T16:04:47.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b56fc19a-5b8d-4522-981f-d88ff8769ae9 -i td/osd-recovery-prio/1/new.json 2024-09-16T16:04:48.114 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:04:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-09-16T16:04:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfV+hmZRE6HhAA0k3uRK/+J2Y0UMFTqqQmAQ== --osd-uuid b56fc19a-5b8d-4522-981f-d88ff8769ae9 2024-09-16T16:04:48.173 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:48.170+0000 7fba037667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:48.175 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:48.174+0000 7fba037667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:48.177 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:48.174+0000 7fba037667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:48.177 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:48.174+0000 7fba037667c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-16T16:04:48.177 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:48.174+0000 7fba037667c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-16T16:04:48.177 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:48.174+0000 7fba037667c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-16T16:04:48.256 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:48.254+0000 7fba037667c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-09-16T16:04:53.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-09-16T16:04:53.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:04:53.885 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:04:53.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:04:53.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:04:54.454 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:04:54.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:04:54.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:04:54.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:04:54.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:04:54.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:04:54.490 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:54.490+0000 7efea83447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:54.503 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:54.502+0000 7efea83447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:54.504 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:54.502+0000 7efea83447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:04:54.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:04:54.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:04:54.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:04:54.916 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:04:54.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:04:54.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:04:54.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:54.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:04:54.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:54.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:04:55.087 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:55.086+0000 7efea83447c0 -1 Falling back to public interface 2024-09-16T16:04:55.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:56.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:56.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:56.376 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:04:56.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:04:56.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:56.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:04:56.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:57.754 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:04:57.754+0000 7efea83447c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:04:57.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:57.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:57.802 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:04:57.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:04:57.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:04:57.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:58.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:04:59.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:04:59.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:04:59.346 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:04:59.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:04:59.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:04:59.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:04:59.772 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2734166283,v1:127.0.0.1:6811/2734166283] [v2:127.0.0.1:6812/2734166283,v1:127.0.0.1:6813/2734166283] exists,up b56fc19a-5b8d-4522-981f-d88ff8769ae9 2024-09-16T16:04:59.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:04:59.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:04:59.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:04:59.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: seq 1 3 2024-09-16T16:04:59.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-16T16:04:59.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test1 1 1 2024-09-16T16:04:59.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-09-16T16:05:00.365 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test1' already exists 2024-09-16T16:05:00.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:05:01.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-09-16T16:05:02.299 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-09-16T16:05:02.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-16T16:05:02.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test2 1 1 2024-09-16T16:05:02.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-09-16T16:05:03.045 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test2' already exists 2024-09-16T16:05:03.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:05:04.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-09-16T16:05:04.868 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 2 size to 2 2024-09-16T16:05:04.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-16T16:05:04.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test3 1 1 2024-09-16T16:05:04.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-09-16T16:05:05.563 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test3' already exists 2024-09-16T16:05:05.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:05:06.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2024-09-16T16:05:07.514 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 3 size to 2 2024-09-16T16:05:07.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:344: TEST_recovery_pool_priority: sleep 5 2024-09-16T16:05:12.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:346: TEST_recovery_pool_priority: wait_for_clean 2024-09-16T16:05:12.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:05:12.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:05:12.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:05:12.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:05:12.543 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:05:12.543 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:05:12.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:05:12.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:05:12.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:05:12.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:05:12.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:05:12.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:05:12.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:05:12.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:05:12.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:05:13.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:05:13.090 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-09-16T16:05:13.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:05:13.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:05:13.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:05:13.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836488 2024-09-16T16:05:13.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836488 2024-09-16T16:05:13.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836488' 2024-09-16T16:05:13.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:05:13.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:05:13.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672966 2024-09-16T16:05:13.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672966 2024-09-16T16:05:13.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2024-09-16T16:05:13.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:05:13.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836488 2024-09-16T16:05:13.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:05:13.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:05:13.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836488 2024-09-16T16:05:13.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:05:13.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836488 2024-09-16T16:05:13.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-16T16:05:13.404 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836488 2024-09-16T16:05:13.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:05:13.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-16T16:05:13.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:05:14.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:05:14.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:05:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-16T16:05:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:05:15.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672966 2024-09-16T16:05:15.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:05:15.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:05:15.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672966 2024-09-16T16:05:15.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:05:15.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672966 2024-09-16T16:05:15.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-16T16:05:15.265 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672966 2024-09-16T16:05:15.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:05:15.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672967 -lt 42949672966 2024-09-16T16:05:15.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:05:15.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:05:15.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:05:16.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-09-16T16:05:16.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:05:16.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:05:16.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:05:16.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:05:16.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:05:16.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:05:16.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:05:16.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-16T16:05:16.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:05:16.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:05:16.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:05:17.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-09-16T16:05:17.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:05:17.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:05:17.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:348: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-16T16:05:17.564 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T16:05:17.564 INFO:tasks.workunit.client.0.smithi039.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:05:07.681555+0000 0'0 26:17 [1,0] 1 [1,0] 1 0'0 2024-09-16T16:05:05.312285+0000 0'0 2024-09-16T16:05:05.312285+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:34:50.280564+0000 0 0 2024-09-16T16:05:17.564 INFO:tasks.workunit.client.0.smithi039.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:05:05.864052+0000 0'0 25:23 [0,1] 0 [0,1] 0 0'0 2024-09-16T16:05:02.727882+0000 0'0 2024-09-16T16:05:02.727882+0000 0 0 periodic scrub scheduled @ 2024-09-17T21:02:09.434235+0000 0 0 2024-09-16T16:05:17.564 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:05:02.842483+0000 0'0 25:26 [1,0] 1 [1,0] 1 0'0 2024-09-16T16:05:00.143127+0000 0'0 2024-09-16T16:05:00.143127+0000 0 0 periodic scrub scheduled @ 2024-09-18T01:49:16.336481+0000 0 0 2024-09-16T16:05:17.564 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:05:17.565 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T16:05:17.565 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:05:17.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local PG1 2024-09-16T16:05:17.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local POOLNUM1 2024-09-16T16:05:17.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:354: TEST_recovery_pool_priority: local pool1 2024-09-16T16:05:17.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local chk_osd1_1 2024-09-16T16:05:17.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local chk_osd1_2 2024-09-16T16:05:17.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local PG2 2024-09-16T16:05:17.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local POOLNUM2 2024-09-16T16:05:17.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:360: TEST_recovery_pool_priority: local pool2 2024-09-16T16:05:17.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: local chk_osd2_1 2024-09-16T16:05:17.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:362: TEST_recovery_pool_priority: local chk_osd2_2 2024-09-16T16:05:17.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: seq 1 3 2024-09-16T16:05:17.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-16T16:05:17.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2024-09-16T16:05:17.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-09-16T16:05:17.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-09-16T16:05:17.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=1 2024-09-16T16:05:17.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-09-16T16:05:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=0 2024-09-16T16:05:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z '' ']' 2024-09-16T16:05:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: PG1=1.0 2024-09-16T16:05:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: POOLNUM1=1 2024-09-16T16:05:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: pool1=test1 2024-09-16T16:05:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:374: TEST_recovery_pool_priority: chk_osd1_1=1 2024-09-16T16:05:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: chk_osd1_2=0 2024-09-16T16:05:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-16T16:05:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2024-09-16T16:05:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-09-16T16:05:18.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-09-16T16:05:18.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=0 2024-09-16T16:05:18.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-09-16T16:05:18.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=1 2024-09-16T16:05:18.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2024-09-16T16:05:18.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2024-09-16T16:05:18.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: PG2=2.0 2024-09-16T16:05:18.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: POOLNUM2=2 2024-09-16T16:05:18.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: pool2=test2 2024-09-16T16:05:18.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:381: TEST_recovery_pool_priority: chk_osd2_1=0 2024-09-16T16:05:18.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:382: TEST_recovery_pool_priority: chk_osd2_2=1 2024-09-16T16:05:18.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: break 2024-09-16T16:05:18.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:386: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2024-09-16T16:05:18.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:388: TEST_recovery_pool_priority: '[' test2 = '' ']' 2024-09-16T16:05:18.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: seq 1 3 2024-09-16T16:05:18.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-16T16:05:18.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2024-09-16T16:05:18.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-16T16:05:18.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2024-09-16T16:05:18.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-16T16:05:18.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2024-09-16T16:05:18.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:398: TEST_recovery_pool_priority: delete_pool test3 2024-09-16T16:05:18.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-09-16T16:05:18.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-09-16T16:05:19.073 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test3' does not exist 2024-09-16T16:05:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool1_extra_prio=1 2024-09-16T16:05:19.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:403: TEST_recovery_pool_priority: pool2_extra_prio=2 2024-09-16T16:05:19.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: expr 190 + 1 2024-09-16T16:05:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: pool1_prio=191 2024-09-16T16:05:19.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: expr 190 + 2 2024-09-16T16:05:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: pool2_prio=192 2024-09-16T16:05:19.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-09-16T16:05:19.899 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 1 2024-09-16T16:05:19.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2024-09-16T16:05:20.732 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 recovery_priority to 1 2024-09-16T16:05:20.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:409: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2024-09-16T16:05:21.578 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 2 size to 1 2024-09-16T16:05:21.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2024-09-16T16:05:22.390 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 2 recovery_priority to 2 2024-09-16T16:05:22.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: wait_for_clean 2024-09-16T16:05:22.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:05:22.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:05:22.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:05:22.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:05:22.412 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:05:22.412 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:05:22.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:05:22.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:05:22.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:05:22.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:05:22.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:05:22.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:05:22.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:05:22.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:05:22.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:05:23.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:05:23.098 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-09-16T16:05:23.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:05:23.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:05:23.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:05:23.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836492 2024-09-16T16:05:23.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836492 2024-09-16T16:05:23.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836492' 2024-09-16T16:05:23.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:05:23.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:05:23.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672970 2024-09-16T16:05:23.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672970 2024-09-16T16:05:23.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836492 1-42949672970' 2024-09-16T16:05:23.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:05:23.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836492 2024-09-16T16:05:23.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:05:23.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:05:23.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836492 2024-09-16T16:05:23.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:05:23.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836492 2024-09-16T16:05:23.440 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836492 2024-09-16T16:05:23.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-16T16:05:23.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:05:23.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836490 -lt 21474836492 2024-09-16T16:05:23.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:05:24.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:05:24.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:05:25.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-16T16:05:25.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:05:25.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672970 2024-09-16T16:05:25.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:05:25.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:05:25.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672970 2024-09-16T16:05:25.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:05:25.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672970 2024-09-16T16:05:25.366 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672970 2024-09-16T16:05:25.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-16T16:05:25.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:05:25.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672971 -lt 42949672970 2024-09-16T16:05:25.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:05:25.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:05:25.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:05:26.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-16T16:05:26.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:05:26.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:05:26.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:05:26.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:05:26.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:05:26.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:05:26.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:05:26.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:05:26.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:05:26.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:05:26.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:05:27.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-16T16:05:27.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:05:27.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:05:27.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:413: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-09-16T16:05:27.379 INFO:tasks.workunit.client.0.smithi039.stderr:10+0 records in 2024-09-16T16:05:27.380 INFO:tasks.workunit.client.0.smithi039.stderr:10+0 records out 2024-09-16T16:05:27.380 INFO:tasks.workunit.client.0.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0394866 s, 266 MB/s 2024-09-16T16:05:27.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: p=1 2024-09-16T16:05:27.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-09-16T16:05:27.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-09-16T16:05:27.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:27.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-09-16T16:05:27.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:27.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-09-16T16:05:28.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:28.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-09-16T16:05:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:28.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-09-16T16:05:28.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:28.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-09-16T16:05:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:29.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-09-16T16:05:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:29.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-09-16T16:05:30.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:30.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-09-16T16:05:30.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:30.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-09-16T16:05:30.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:30.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-09-16T16:05:31.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:31.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-09-16T16:05:31.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:31.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-09-16T16:05:31.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:31.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-09-16T16:05:32.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:32.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-09-16T16:05:32.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:32.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-09-16T16:05:33.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:33.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-09-16T16:05:33.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:33.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-09-16T16:05:33.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:33.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-09-16T16:05:34.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:34.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-09-16T16:05:34.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:34.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-09-16T16:05:34.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:34.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-09-16T16:05:35.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:35.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-09-16T16:05:35.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:35.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-09-16T16:05:36.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:36.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-09-16T16:05:36.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:36.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-09-16T16:05:36.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:36.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-09-16T16:05:37.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:37.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-09-16T16:05:37.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:37.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-09-16T16:05:38.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:38.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-09-16T16:05:38.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:38.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-09-16T16:05:38.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:38.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-09-16T16:05:39.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:39.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-09-16T16:05:39.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-09-16T16:05:40.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:40.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-09-16T16:05:40.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:40.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-09-16T16:05:40.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:40.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-09-16T16:05:41.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:41.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-09-16T16:05:41.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:41.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-09-16T16:05:41.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:41.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-09-16T16:05:42.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:42.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-09-16T16:05:42.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:42.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-09-16T16:05:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:43.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-09-16T16:05:43.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:43.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-09-16T16:05:43.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:43.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-09-16T16:05:44.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:44.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-09-16T16:05:44.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:44.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-09-16T16:05:44.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:44.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-09-16T16:05:45.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:45.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-09-16T16:05:45.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:45.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-09-16T16:05:46.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:46.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-09-16T16:05:46.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:46.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-09-16T16:05:46.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:46.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-09-16T16:05:47.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:47.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-09-16T16:05:47.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:47.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-09-16T16:05:47.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:47.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-09-16T16:05:48.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:48.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-09-16T16:05:48.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:48.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-09-16T16:05:49.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:49.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-09-16T16:05:49.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:49.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-09-16T16:05:49.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:49.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-09-16T16:05:50.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:50.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-09-16T16:05:50.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:50.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-09-16T16:05:50.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:50.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-09-16T16:05:51.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:51.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-09-16T16:05:51.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:51.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-09-16T16:05:52.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:52.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-09-16T16:05:52.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:52.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-09-16T16:05:52.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:52.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-09-16T16:05:53.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:53.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-09-16T16:05:53.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:53.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-09-16T16:05:54.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:54.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-09-16T16:05:54.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:54.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-09-16T16:05:54.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:54.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-09-16T16:05:55.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:55.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-09-16T16:05:55.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:55.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-09-16T16:05:55.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:55.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-09-16T16:05:56.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:56.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-09-16T16:05:56.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:56.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-09-16T16:05:56.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:56.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-09-16T16:05:57.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:57.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-09-16T16:05:57.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:57.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-09-16T16:05:58.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:58.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-09-16T16:05:58.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:58.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-09-16T16:05:59.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:59.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-09-16T16:05:59.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:59.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-09-16T16:05:59.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:05:59.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-09-16T16:06:00.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:00.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-09-16T16:06:00.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:00.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-09-16T16:06:00.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:00.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-09-16T16:06:01.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:01.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-09-16T16:06:01.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:01.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-09-16T16:06:02.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:02.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-09-16T16:06:02.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:02.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-09-16T16:06:02.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:02.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-09-16T16:06:03.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:03.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-09-16T16:06:03.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:03.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-09-16T16:06:04.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:04.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-09-16T16:06:04.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:04.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-09-16T16:06:04.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:04.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-09-16T16:06:05.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:05.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-09-16T16:06:05.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:05.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-09-16T16:06:06.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:06.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-09-16T16:06:06.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:06.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-09-16T16:06:06.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:06.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-09-16T16:06:07.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:07.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-09-16T16:06:07.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:07.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-09-16T16:06:08.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:08.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-09-16T16:06:08.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:08.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-09-16T16:06:08.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:08.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-09-16T16:06:09.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:09.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-09-16T16:06:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:09.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-09-16T16:06:10.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:10.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-09-16T16:06:10.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:10.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-09-16T16:06:10.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:10.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-09-16T16:06:11.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:11.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-09-16T16:06:11.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:11.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-09-16T16:06:12.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:12.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-09-16T16:06:12.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:12.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-09-16T16:06:12.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:12.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-09-16T16:06:13.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:13.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-09-16T16:06:13.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:13.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-09-16T16:06:14.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:14.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-09-16T16:06:14.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:14.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-09-16T16:06:14.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:14.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-09-16T16:06:15.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:15.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-09-16T16:06:15.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:15.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-09-16T16:06:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:15.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-09-16T16:06:16.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:16.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-09-16T16:06:16.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:16.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-09-16T16:06:16.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:16.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-09-16T16:06:17.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:17.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-09-16T16:06:17.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:17.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-09-16T16:06:18.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:18.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-09-16T16:06:18.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:18.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-09-16T16:06:18.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:18.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-09-16T16:06:19.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:19.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-09-16T16:06:19.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:19.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-09-16T16:06:20.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:20.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-09-16T16:06:20.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:20.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-09-16T16:06:20.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:20.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-09-16T16:06:21.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:21.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-09-16T16:06:21.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:21.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-09-16T16:06:21.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:21.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-09-16T16:06:22.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:22.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-09-16T16:06:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-09-16T16:06:22.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:22.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-09-16T16:06:23.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:23.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-09-16T16:06:23.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:23.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-09-16T16:06:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:24.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-09-16T16:06:24.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:24.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-09-16T16:06:24.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:24.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-09-16T16:06:25.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:25.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-09-16T16:06:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:25.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-09-16T16:06:26.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:26.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-09-16T16:06:26.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:26.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-09-16T16:06:26.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:26.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-09-16T16:06:27.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:27.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-09-16T16:06:27.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:27.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-09-16T16:06:28.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:28.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-09-16T16:06:28.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:28.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-09-16T16:06:28.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:28.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-09-16T16:06:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-09-16T16:06:29.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:29.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-09-16T16:06:30.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:30.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-09-16T16:06:30.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:30.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-09-16T16:06:30.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:30.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-09-16T16:06:31.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:31.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-09-16T16:06:31.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:31.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-09-16T16:06:31.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:31.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-09-16T16:06:32.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:32.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-09-16T16:06:32.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:32.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-09-16T16:06:33.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:33.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-09-16T16:06:33.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:33.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-09-16T16:06:33.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:33.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-09-16T16:06:34.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:34.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-09-16T16:06:34.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:34.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-09-16T16:06:34.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:34.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-09-16T16:06:35.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:35.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-09-16T16:06:35.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:35.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-09-16T16:06:36.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:36.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-09-16T16:06:36.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:36.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-09-16T16:06:36.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:36.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-09-16T16:06:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:37.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-09-16T16:06:37.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:37.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-09-16T16:06:38.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:38.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-09-16T16:06:38.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:38.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-09-16T16:06:38.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:38.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-09-16T16:06:39.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:39.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-09-16T16:06:39.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:39.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-09-16T16:06:40.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:40.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-09-16T16:06:40.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:40.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-09-16T16:06:40.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:40.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-09-16T16:06:41.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:41.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-09-16T16:06:41.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:41.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-09-16T16:06:42.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:42.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-09-16T16:06:42.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:42.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-09-16T16:06:42.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:42.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-09-16T16:06:43.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:43.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-09-16T16:06:43.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:43.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-09-16T16:06:43.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:43.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-09-16T16:06:44.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 1 + 1 2024-09-16T16:06:44.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=2 2024-09-16T16:06:44.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-09-16T16:06:44.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-09-16T16:06:44.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:44.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2024-09-16T16:06:44.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:44.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2024-09-16T16:06:45.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:45.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2024-09-16T16:06:45.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:45.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2024-09-16T16:06:45.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:45.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2024-09-16T16:06:46.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:46.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2024-09-16T16:06:46.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:46.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2024-09-16T16:06:46.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:46.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2024-09-16T16:06:47.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:47.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2024-09-16T16:06:47.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:47.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2024-09-16T16:06:48.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:48.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2024-09-16T16:06:48.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:48.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2024-09-16T16:06:48.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:48.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2024-09-16T16:06:49.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:49.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2024-09-16T16:06:49.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:49.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2024-09-16T16:06:50.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:50.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2024-09-16T16:06:50.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:50.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2024-09-16T16:06:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:50.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2024-09-16T16:06:51.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:51.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2024-09-16T16:06:51.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:51.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2024-09-16T16:06:52.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:52.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2024-09-16T16:06:52.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:52.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2024-09-16T16:06:52.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:52.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2024-09-16T16:06:53.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:53.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2024-09-16T16:06:53.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:53.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2024-09-16T16:06:54.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:54.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2024-09-16T16:06:54.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:54.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2024-09-16T16:06:54.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:54.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2024-09-16T16:06:55.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:55.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2024-09-16T16:06:55.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:55.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2024-09-16T16:06:56.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:56.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2024-09-16T16:06:56.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:56.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2024-09-16T16:06:56.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:56.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2024-09-16T16:06:57.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:57.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2024-09-16T16:06:57.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:57.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2024-09-16T16:06:58.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:58.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2024-09-16T16:06:58.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:58.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2024-09-16T16:06:58.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:58.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2024-09-16T16:06:59.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:59.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2024-09-16T16:06:59.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:59.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2024-09-16T16:06:59.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:06:59.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2024-09-16T16:07:00.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:00.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2024-09-16T16:07:00.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:00.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2024-09-16T16:07:01.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:01.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2024-09-16T16:07:01.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:01.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2024-09-16T16:07:01.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:01.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2024-09-16T16:07:02.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:02.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2024-09-16T16:07:02.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:02.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2024-09-16T16:07:03.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:03.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2024-09-16T16:07:03.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:03.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2024-09-16T16:07:03.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:03.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2024-09-16T16:07:04.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:04.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2024-09-16T16:07:04.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:04.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2024-09-16T16:07:04.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:04.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2024-09-16T16:07:05.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:05.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2024-09-16T16:07:05.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:05.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2024-09-16T16:07:06.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:06.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2024-09-16T16:07:06.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:06.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2024-09-16T16:07:06.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:06.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2024-09-16T16:07:07.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:07.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2024-09-16T16:07:07.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:07.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2024-09-16T16:07:08.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:08.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2024-09-16T16:07:08.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:08.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2024-09-16T16:07:08.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:08.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2024-09-16T16:07:09.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:09.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2024-09-16T16:07:09.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:09.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2024-09-16T16:07:10.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:10.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2024-09-16T16:07:10.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:10.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2024-09-16T16:07:10.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:10.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2024-09-16T16:07:11.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:11.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2024-09-16T16:07:11.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:11.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2024-09-16T16:07:12.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:12.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2024-09-16T16:07:12.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:12.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2024-09-16T16:07:12.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:12.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2024-09-16T16:07:13.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:13.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2024-09-16T16:07:13.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:13.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2024-09-16T16:07:14.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:14.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2024-09-16T16:07:14.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:14.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2024-09-16T16:07:14.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:14.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2024-09-16T16:07:15.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:15.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2024-09-16T16:07:15.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:15.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2024-09-16T16:07:16.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:16.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2024-09-16T16:07:16.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:16.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2024-09-16T16:07:16.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:16.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2024-09-16T16:07:17.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:17.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2024-09-16T16:07:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:17.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2024-09-16T16:07:18.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:18.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2024-09-16T16:07:18.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:18.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2024-09-16T16:07:18.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:18.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2024-09-16T16:07:19.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:19.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2024-09-16T16:07:19.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:19.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2024-09-16T16:07:20.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:20.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2024-09-16T16:07:20.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:20.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2024-09-16T16:07:20.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:20.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2024-09-16T16:07:21.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:21.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2024-09-16T16:07:21.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:21.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2024-09-16T16:07:22.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:22.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2024-09-16T16:07:22.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:22.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2024-09-16T16:07:22.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:22.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2024-09-16T16:07:23.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:23.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2024-09-16T16:07:23.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:23.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2024-09-16T16:07:24.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:24.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2024-09-16T16:07:24.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:24.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2024-09-16T16:07:24.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:24.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2024-09-16T16:07:25.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:25.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2024-09-16T16:07:25.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:25.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2024-09-16T16:07:26.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:26.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2024-09-16T16:07:26.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:26.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2024-09-16T16:07:26.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:26.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2024-09-16T16:07:27.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:27.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2024-09-16T16:07:27.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:27.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2024-09-16T16:07:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2024-09-16T16:07:28.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:28.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2024-09-16T16:07:28.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:28.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2024-09-16T16:07:29.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:29.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2024-09-16T16:07:29.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:29.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2024-09-16T16:07:30.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:30.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2024-09-16T16:07:30.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:30.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2024-09-16T16:07:30.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:30.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2024-09-16T16:07:31.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:31.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2024-09-16T16:07:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:31.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2024-09-16T16:07:32.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:32.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2024-09-16T16:07:32.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:32.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2024-09-16T16:07:32.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:32.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2024-09-16T16:07:33.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:33.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2024-09-16T16:07:33.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:33.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2024-09-16T16:07:34.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:34.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2024-09-16T16:07:34.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:34.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2024-09-16T16:07:34.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:34.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2024-09-16T16:07:35.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:35.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2024-09-16T16:07:35.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:35.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2024-09-16T16:07:35.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:35.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2024-09-16T16:07:36.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:36.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2024-09-16T16:07:36.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:36.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2024-09-16T16:07:36.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:36.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2024-09-16T16:07:37.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:37.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2024-09-16T16:07:37.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:37.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2024-09-16T16:07:38.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:38.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2024-09-16T16:07:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:38.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2024-09-16T16:07:38.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:38.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2024-09-16T16:07:39.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:39.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2024-09-16T16:07:39.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:39.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2024-09-16T16:07:40.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:40.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2024-09-16T16:07:40.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:40.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2024-09-16T16:07:41.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:41.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2024-09-16T16:07:41.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:41.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2024-09-16T16:07:41.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:41.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2024-09-16T16:07:42.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:42.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2024-09-16T16:07:42.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:42.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2024-09-16T16:07:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2024-09-16T16:07:43.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:43.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2024-09-16T16:07:43.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:43.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2024-09-16T16:07:44.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:44.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2024-09-16T16:07:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:44.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2024-09-16T16:07:45.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:45.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2024-09-16T16:07:45.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:45.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2024-09-16T16:07:45.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:45.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2024-09-16T16:07:46.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:46.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2024-09-16T16:07:46.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:46.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2024-09-16T16:07:47.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:47.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2024-09-16T16:07:47.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:47.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2024-09-16T16:07:47.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:47.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2024-09-16T16:07:48.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:48.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2024-09-16T16:07:48.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:48.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2024-09-16T16:07:49.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:49.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2024-09-16T16:07:49.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:49.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2024-09-16T16:07:49.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:49.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2024-09-16T16:07:50.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:50.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2024-09-16T16:07:50.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:50.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2024-09-16T16:07:51.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:51.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2024-09-16T16:07:51.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:51.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2024-09-16T16:07:51.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:51.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2024-09-16T16:07:52.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:52.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2024-09-16T16:07:52.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:52.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2024-09-16T16:07:53.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:53.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2024-09-16T16:07:53.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:53.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2024-09-16T16:07:53.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:53.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2024-09-16T16:07:54.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:54.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2024-09-16T16:07:54.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:54.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2024-09-16T16:07:55.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:55.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2024-09-16T16:07:55.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:55.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2024-09-16T16:07:56.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:56.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2024-09-16T16:07:56.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:56.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2024-09-16T16:07:56.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:56.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2024-09-16T16:07:57.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:57.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2024-09-16T16:07:57.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:57.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2024-09-16T16:07:57.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:57.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2024-09-16T16:07:58.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:58.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2024-09-16T16:07:58.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:58.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2024-09-16T16:07:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2024-09-16T16:07:59.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:59.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2024-09-16T16:07:59.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:07:59.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2024-09-16T16:08:00.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:08:00.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2024-09-16T16:08:00.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:08:00.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2024-09-16T16:08:01.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:08:01.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2024-09-16T16:08:01.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:08:01.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2024-09-16T16:08:02.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:08:02.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2024-09-16T16:08:02.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:08:02.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2024-09-16T16:08:02.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:08:02.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2024-09-16T16:08:03.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-16T16:08:03.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2024-09-16T16:08:03.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 2 + 1 2024-09-16T16:08:03.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=3 2024-09-16T16:08:03.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2024-09-16T16:08:03.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-09-16T16:08:03.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-09-16T16:08:03.637 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-09-16T16:08:03.637 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-09-16T16:08:03.637 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-09-16T16:08:03.637 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-09-16T16:08:03.637 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:08:04.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-16T16:08:04.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-09-16T16:08:04.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-16T16:08:04.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: local otherosd=null 2024-09-16T16:08:04.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-16T16:08:04.966 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T16:08:04.967 INFO:tasks.workunit.client.0.smithi039.stdout:2.0 199 0 0 0 0 2080374784 0 0 595 0 595 active+clean 2024-09-16T16:05:21.743550+0000 35'595 35:633 [0] 0 [0] 0 0'0 2024-09-16T16:05:02.727882+0000 0'0 2024-09-16T16:05:02.727882+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:50:44.118538+0000 0 0 2024-09-16T16:08:04.967 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-16T16:05:20.101554+0000 35'600 35:642 [1] 1 [1] 1 0'0 2024-09-16T16:05:00.143127+0000 0'0 2024-09-16T16:05:00.143127+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:14:42.623204+0000 0 0 2024-09-16T16:08:04.967 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:08:04.967 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T16:08:04.967 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:08:04.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ERRORS=0 2024-09-16T16:08:04.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-09-16T16:08:05.884 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-09-16T16:08:05.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-09-16T16:08:06.695 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 2 size to 2 2024-09-16T16:08:06.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-16T16:08:07.148 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T16:08:07.148 INFO:tasks.workunit.client.0.smithi039.stdout:2.0 199 0 0 0 0 2080374784 0 0 595 0 595 active+clean 2024-09-16T16:05:21.743550+0000 35'595 35:633 [0] 0 [0] 0 0'0 2024-09-16T16:05:02.727882+0000 0'0 2024-09-16T16:05:02.727882+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:50:44.118538+0000 0 0 2024-09-16T16:08:07.148 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-16T16:05:20.101554+0000 35'600 35:642 [1] 1 [1] 1 0'0 2024-09-16T16:05:00.143127+0000 0'0 2024-09-16T16:05:00.143127+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:14:42.623204+0000 0 0 2024-09-16T16:08:07.148 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:08:07.148 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T16:08:07.149 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:08:07.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:436: TEST_recovery_pool_priority: set -o pipefail 2024-09-16T16:08:07.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: count=0 2024-09-16T16:08:07.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-09-16T16:08:07.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-09-16T16:08:07.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-09-16T16:08:07.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-09-16T16:08:07.661 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:08:07.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 0 == 2 2024-09-16T16:08:07.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-09-16T16:08:09.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 0 -eq 10 2024-09-16T16:08:09.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 0 + 1 2024-09-16T16:08:09.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=1 2024-09-16T16:08:09.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-09-16T16:08:09.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-09-16T16:08:09.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-09-16T16:08:09.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-09-16T16:08:10.115 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:08:10.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 1 == 2 2024-09-16T16:08:10.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-09-16T16:08:12.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 1 -eq 10 2024-09-16T16:08:12.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 1 + 1 2024-09-16T16:08:12.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=2 2024-09-16T16:08:12.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-09-16T16:08:12.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-09-16T16:08:12.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-09-16T16:08:12.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-09-16T16:08:12.552 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:08:12.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 2 == 2 2024-09-16T16:08:12.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: break 2024-09-16T16:08:12.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: set +o pipefail 2024-09-16T16:08:12.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-16T16:08:12.972 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T16:08:12.972 INFO:tasks.workunit.client.0.smithi039.stdout:2.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-09-16T16:08:07.659737+0000 35'600 41:652 [0,1] 0 [0] 0 0'0 2024-09-16T16:05:02.727882+0000 0'0 2024-09-16T16:05:02.727882+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:16:34.552846+0000 0 0 2024-09-16T16:08:12.972 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 200 0 199 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-09-16T16:08:07.634961+0000 35'600 41:656 [1,0] 1 [1] 1 0'0 2024-09-16T16:05:00.143127+0000 0'0 2024-09-16T16:05:00.143127+0000 0 0 periodic scrub scheduled @ 2024-09-17T19:11:12.672594+0000 0 0 2024-09-16T16:08:12.972 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:08:12.972 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T16:08:12.972 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:08:12.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.1 2024-09-16T16:08:12.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T16:08:12.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T16:08:12.990 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:08:12.990 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:08:12.990 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:08:12.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.1.asok 2024-09-16T16:08:12.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2024-09-16T16:08:12.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:08:13.105 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 2024-09-16T16:08:13.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.1 2024-09-16T16:08:13.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-16T16:08:13.106 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:08:13.106 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:08:13.106 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:08:13.106 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:08:13.106 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:08:13.106 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:08:13.106 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 191, 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "2.0", 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 192, 2024-09-16T16:08:13.107 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:08:13.108 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:08:13.108 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:08:13.108 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:08:13.108 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:08:13.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: get_asok_path osd.0 2024-09-16T16:08:13.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T16:08:13.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T16:08:13.109 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:08:13.109 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:08:13.109 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:08:13.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.0.asok 2024-09-16T16:08:13.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: CEPH_ARGS= 2024-09-16T16:08:13.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T16:08:13.218 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 2024-09-16T16:08:13.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:460: TEST_recovery_pool_priority: echo osd.0 2024-09-16T16:08:13.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-16T16:08:13.220 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:08:13.220 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:08:13.220 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:08:13.220 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:08:13.220 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:08:13.220 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:08:13.220 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:08:13.220 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "2.0", 2024-09-16T16:08:13.220 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 192, 2024-09-16T16:08:13.220 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:08:13.220 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 191, 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:08:13.221 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:08:13.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-16T16:08:13.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-09-16T16:08:13.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-09-16T16:08:13.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: ITEM=1.0 2024-09-16T16:08:13.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:465: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-09-16T16:08:13.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-16T16:08:13.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-16T16:08:13.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: PRIO=191 2024-09-16T16:08:13.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:471: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-09-16T16:08:13.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-16T16:08:13.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-09-16T16:08:13.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-09-16T16:08:13.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: ITEM=1.0 2024-09-16T16:08:13.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:480: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-09-16T16:08:13.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-16T16:08:13.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-09-16T16:08:13.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: PRIO=191 2024-09-16T16:08:13.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:486: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-09-16T16:08:13.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-16T16:08:13.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-09-16T16:08:13.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-09-16T16:08:13.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: ITEM=2.0 2024-09-16T16:08:13.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:495: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-09-16T16:08:13.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-16T16:08:13.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-16T16:08:13.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: PRIO=192 2024-09-16T16:08:13.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:501: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-09-16T16:08:13.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-16T16:08:13.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-09-16T16:08:13.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-09-16T16:08:13.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: ITEM=2.0 2024-09-16T16:08:13.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:510: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-09-16T16:08:13.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-09-16T16:08:13.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-16T16:08:13.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: PRIO=192 2024-09-16T16:08:13.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:516: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-09-16T16:08:13.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:523: TEST_recovery_pool_priority: wait_for_clean 2024-09-16T16:08:13.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:08:13.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:08:13.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:08:13.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:08:13.518 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:08:13.518 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:08:13.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:08:13.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:08:13.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:08:13.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:08:13.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:08:13.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:08:13.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:08:13.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:08:13.622 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:08:14.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:08:14.182 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-09-16T16:08:14.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:08:14.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:08:14.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:08:14.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836528 2024-09-16T16:08:14.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836528 2024-09-16T16:08:14.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836528' 2024-09-16T16:08:14.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:08:14.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:08:14.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673007 2024-09-16T16:08:14.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673007 2024-09-16T16:08:14.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836528 1-42949673007' 2024-09-16T16:08:14.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:08:14.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836528 2024-09-16T16:08:14.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:08:14.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:08:14.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836528 2024-09-16T16:08:14.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:08:14.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836528 2024-09-16T16:08:14.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2024-09-16T16:08:14.625 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836528 2024-09-16T16:08:14.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:08:15.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836526 -lt 21474836528 2024-09-16T16:08:15.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:08:16.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:08:16.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:08:16.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836528 -lt 21474836528 2024-09-16T16:08:16.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:08:16.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673007 2024-09-16T16:08:16.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:08:16.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:08:16.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673007 2024-09-16T16:08:16.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:08:16.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673007 2024-09-16T16:08:16.612 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673007 2024-09-16T16:08:16.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2024-09-16T16:08:16.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:08:17.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673007 -lt 42949673007 2024-09-16T16:08:17.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:08:17.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:17.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:18.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-16T16:08:18.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:18.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:18.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:18.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:18.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:18.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:18.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:18.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:18.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:18.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:18.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:19.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:19.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T16:08:19.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T16:08:19.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T16:08:19.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:19.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:19.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:19.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:19.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:19.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:19.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:19.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:19.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:19.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:19.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:19.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:19.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:19.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:20.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:20.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:20.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:20.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:20.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:20.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:20.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:20.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:20.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:21.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18674095 2024-09-16T16:08:21.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18674095 '!=' null 2024-09-16T16:08:21.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:21.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:21.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:21.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:21.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:21.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:21.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:21.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:21.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:21.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:21.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:21.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:21.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:21.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:21.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:22.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:22.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:22.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:22.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:22.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:22.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:22.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:22.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:22.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586616 2024-09-16T16:08:22.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586616 '!=' null 2024-09-16T16:08:22.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:22.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:22.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:23.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:23.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:23.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:23.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:23.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:23.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:23.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:23.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:23.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:23.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:23.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:23.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:24.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:24.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:24.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:24.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:24.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:24.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:24.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:24.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:24.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:24.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954595 2024-09-16T16:08:24.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954595 '!=' null 2024-09-16T16:08:24.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:24.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:24.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:25.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:25.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:25.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:25.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:25.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:25.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:25.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:25.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:25.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:25.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:25.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:25.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:26.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:26.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:26.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:26.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:26.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:26.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:26.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:26.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:26.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:26.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828389 2024-09-16T16:08:26.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828389 '!=' null 2024-09-16T16:08:26.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:26.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:26.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:26.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:26.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:26.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:26.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:26.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:26.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:26.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:26.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:27.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:27.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:27.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:27.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:28.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:28.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:28.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:28.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:28.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:28.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:28.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:28.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:28.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20966199 2024-09-16T16:08:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20966199 '!=' null 2024-09-16T16:08:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:28.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:28.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:28.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:28.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:28.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:28.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:28.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:28.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:29.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:29.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:29.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:29.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:30.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:30.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:30.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:30.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:30.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:30.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:30.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:30.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:30.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:30.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27081748 2024-09-16T16:08:30.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27081748 '!=' null 2024-09-16T16:08:30.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:30.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:30.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:30.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:30.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:30.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:30.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:30.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:30.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:30.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:30.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:31.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:31.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:31.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:31.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:31.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:31.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:31.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:31.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:31.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:31.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:31.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:31.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:31.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:32.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828679 2024-09-16T16:08:32.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828679 '!=' null 2024-09-16T16:08:32.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:32.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:32.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:32.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:32.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:32.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:32.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:32.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:32.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:32.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:32.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:33.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:33.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:33.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:33.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:33.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:33.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:33.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:33.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:33.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:33.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:33.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:33.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:33.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:34.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828866 2024-09-16T16:08:34.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828866 '!=' null 2024-09-16T16:08:34.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:34.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:34.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:34.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:34.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:34.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:34.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:34.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:34.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:34.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:34.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:34.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:34.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:34.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:34.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:35.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:35.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:35.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:35.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:35.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:35.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:35.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:35.383 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:35.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:36.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828655 2024-09-16T16:08:36.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828655 '!=' null 2024-09-16T16:08:36.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:36.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:36.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:36.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:36.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:36.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:36.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:36.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:36.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:36.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:36.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:36.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:36.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:36.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:36.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:37.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:37.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:37.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:37.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:37.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:37.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:37.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:37.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:37.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:38.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713396 2024-09-16T16:08:38.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713396 '!=' null 2024-09-16T16:08:38.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:38.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:38.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:38.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:38.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:38.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:38.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:38.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:38.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:38.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:38.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:38.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:38.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:38.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:38.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:39.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:39.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:39.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:39.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:39.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:39.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:39.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:39.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:39.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:40.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828544 2024-09-16T16:08:40.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828544 '!=' null 2024-09-16T16:08:40.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:40.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:40.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:40.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:40.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:40.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:40.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:40.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:40.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:40.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:40.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:40.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:40.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:40.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:40.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:41.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:41.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:41.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:41.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:41.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:41.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:41.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:41.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:41.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:42.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954527 2024-09-16T16:08:42.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954527 '!=' null 2024-09-16T16:08:42.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:42.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:42.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:42.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:42.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:42.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:42.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:42.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:42.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:42.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:42.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:42.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:42.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:42.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:42.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:43.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:43.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:43.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:43.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:43.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:43.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:43.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:43.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:43.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:44.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27081138 2024-09-16T16:08:44.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27081138 '!=' null 2024-09-16T16:08:44.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:44.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:44.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:44.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:44.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:44.233 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:44.233 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:44.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:44.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:44.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:44.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:44.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:44.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:44.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:44.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:45.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:45.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:45.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:45.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:45.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:45.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:45.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:45.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:45.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:46.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954180 2024-09-16T16:08:46.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954180 '!=' null 2024-09-16T16:08:46.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:46.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:46.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:46.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:46.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:46.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:46.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:46.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:46.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:46.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:46.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:46.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:46.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:46.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:46.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:47.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:47.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:47.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:47.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:47.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:47.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:47.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:47.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:47.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712988 2024-09-16T16:08:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712988 '!=' null 2024-09-16T16:08:48.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:48.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:48.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:48.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:48.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:48.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:48.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:48.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:48.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:48.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:48.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:48.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:48.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:48.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:48.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:49.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:49.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:49.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:49.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:49.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:49.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:49.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:49.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:49.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:49.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827695 2024-09-16T16:08:49.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827695 '!=' null 2024-09-16T16:08:49.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:49.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:49.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:49.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:49.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:49.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:49.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:49.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:49.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:49.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:49.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:50.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:50.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:50.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:50.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:51.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:51.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:51.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:51.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:51.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:51.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:51.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:51.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:51.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:51.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27953879 2024-09-16T16:08:51.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27953879 '!=' null 2024-09-16T16:08:51.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:51.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:51.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:51.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:51.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:51.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:51.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:51.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:51.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:51.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:51.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:52.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:52.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:52.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:52.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:53.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:53.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:53.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:53.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:53.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:53.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:53.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:53.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:53.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:53.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954222 2024-09-16T16:08:53.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954222 '!=' null 2024-09-16T16:08:53.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:53.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:53.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:53.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:53.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:53.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:53.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:53.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:53.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:53.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:53.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:54.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:54.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:54.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:54.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:55.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:55.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:55.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:55.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:55.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:55.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:55.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:55.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:55.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:56.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827010 2024-09-16T16:08:56.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827010 '!=' null 2024-09-16T16:08:56.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:56.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:56.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:56.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:56.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:56.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:56.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:56.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:56.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:56.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:56.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:56.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:56.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:56.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:56.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:57.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:57.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:57.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:57.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:57.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:57.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:57.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:57.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:57.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:08:58.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712094 2024-09-16T16:08:58.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712094 '!=' null 2024-09-16T16:08:58.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:08:58.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:08:58.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:08:58.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:08:58.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:08:58.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:08:58.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:08:58.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:08:58.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:08:58.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:08:58.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:08:58.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:08:58.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:08:58.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:08:58.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:08:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:08:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:08:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:08:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:08:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:08:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:08:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:08:59.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:08:59.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:00.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827168 2024-09-16T16:09:00.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827168 '!=' null 2024-09-16T16:09:00.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:00.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:00.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:00.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:00.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:00.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:00.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:00.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:00.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:00.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:00.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:00.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:00.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:00.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:00.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:01.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:01.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:01.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:01.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:01.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827437 2024-09-16T16:09:01.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827437 '!=' null 2024-09-16T16:09:01.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:01.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:01.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:02.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:02.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:02.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:02.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:02.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:02.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:02.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:02.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:02.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:02.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:02.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:02.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:03.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:03.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:03.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:03.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:03.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:03.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:03.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:03.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:03.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:03.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701295 2024-09-16T16:09:03.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701295 '!=' null 2024-09-16T16:09:03.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:03.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:03.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:04.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:04.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:04.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:04.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:04.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:04.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:04.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:04.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:04.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:04.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:04.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:04.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:05.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:05.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:05.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:05.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:05.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:05.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:05.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:05.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:05.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:05.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701401 2024-09-16T16:09:05.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701401 '!=' null 2024-09-16T16:09:05.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:05.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:05.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:05.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:05.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:05.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:05.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:05.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:05.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:05.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:05.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:06.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:06.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:06.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:07.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:07.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:07.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:07.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:07.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:07.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:07.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:07.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:07.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:08.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586895 2024-09-16T16:09:08.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586895 '!=' null 2024-09-16T16:09:08.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:08.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:08.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:08.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:08.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:08.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:08.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:08.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:08.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:08.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:08.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:08.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:08.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:09.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:09.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:09.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:09.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:09.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:09.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:09.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:09.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:09.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828630 2024-09-16T16:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828630 '!=' null 2024-09-16T16:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:10.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:10.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:10.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:10.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:10.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:10.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:10.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:10.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:10.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:10.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:10.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:10.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:11.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:11.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:11.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:11.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:11.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:11.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:11.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:11.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:11.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:11.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828270 2024-09-16T16:09:11.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828270 '!=' null 2024-09-16T16:09:11.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:11.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:11.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:11.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:11.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:11.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:11.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:11.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:11.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:11.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:11.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:12.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:12.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:12.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:12.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:13.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:13.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:13.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:13.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:13.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:13.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:13.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828191 2024-09-16T16:09:13.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828191 '!=' null 2024-09-16T16:09:13.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:13.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:13.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:13.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:13.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:13.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:13.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:13.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:13.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:13.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:13.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:14.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:14.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:14.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:14.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:14.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:14.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:14.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:14.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:14.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:14.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:14.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:14.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:14.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:15.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954557 2024-09-16T16:09:15.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954557 '!=' null 2024-09-16T16:09:15.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:15.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:15.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:15.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:15.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:15.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:15.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:15.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:15.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:15.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:15.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:16.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:16.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:16.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:16.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:16.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:16.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:16.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:16.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:16.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:16.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:16.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:16.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:16.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:17.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954557 2024-09-16T16:09:17.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954557 '!=' null 2024-09-16T16:09:17.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:17.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:17.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:17.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:17.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:17.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:17.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:17.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:17.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:17.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:17.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:17.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:17.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:17.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:17.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:18.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:18.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:18.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:18.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:18.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:18.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:18.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:18.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:18.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:19.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712921 2024-09-16T16:09:19.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712921 '!=' null 2024-09-16T16:09:19.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:19.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:19.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:19.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:19.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:19.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:19.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:19.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:19.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:19.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:19.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:19.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:19.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:19.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:19.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:20.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:20.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:20.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:20.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:20.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:20.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:20.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:20.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:20.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:21.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954284 2024-09-16T16:09:21.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954284 '!=' null 2024-09-16T16:09:21.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:21.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:21.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:21.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:21.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:21.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:21.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:21.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:21.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:21.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:21.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:21.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:21.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:21.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:21.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:22.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:22.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:22.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:22.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:22.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:22.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:22.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:22.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:22.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:23.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827938 2024-09-16T16:09:23.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827938 '!=' null 2024-09-16T16:09:23.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:23.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:23.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:23.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:23.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:23.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:23.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:23.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:23.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:23.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:23.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:23.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:23.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:23.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:23.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:24.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:24.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:24.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:24.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:24.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:24.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:24.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:24.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:24.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:24.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828541 2024-09-16T16:09:24.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828541 '!=' null 2024-09-16T16:09:24.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:24.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:24.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:25.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:25.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:25.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:25.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:25.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:25.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:25.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:25.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:25.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:25.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:25.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:25.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:26.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:26.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:26.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:26.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:26.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:26.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:26.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:26.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:26.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:26.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828699 2024-09-16T16:09:26.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828699 '!=' null 2024-09-16T16:09:26.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:26.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:26.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:26.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:26.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:26.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:26.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:26.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:26.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:26.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:26.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:27.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:27.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:27.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:27.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:28.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:28.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:28.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:28.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:28.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:28.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:28.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:28.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:28.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23587321 2024-09-16T16:09:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23587321 '!=' null 2024-09-16T16:09:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:28.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:28.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:28.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:28.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:28.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:28.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:28.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:28.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:29.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:29.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:29.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:29.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:29.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:29.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:29.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:29.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:29.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:29.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:29.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:29.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:29.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828957 2024-09-16T16:09:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828957 '!=' null 2024-09-16T16:09:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:30.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:30.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:30.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:30.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:30.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:30.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:30.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:30.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:31.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:31.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:31.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:31.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:31.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:31.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:31.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:31.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:31.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:31.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:31.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:31.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:31.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:32.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28829213 2024-09-16T16:09:32.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28829213 '!=' null 2024-09-16T16:09:32.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:32.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:32.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:32.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:32.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:32.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:32.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:32.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:32.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:32.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:32.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:33.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:33.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:33.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:33.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:33.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:33.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:33.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:33.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:34.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28829021 2024-09-16T16:09:34.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28829021 '!=' null 2024-09-16T16:09:34.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:34.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:34.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:34.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:34.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:34.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:34.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:34.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:34.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:34.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:34.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:34.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:34.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:34.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:34.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:35.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:35.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:35.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:35.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:35.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:35.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:35.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:35.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:35.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:36.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828434 2024-09-16T16:09:36.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828434 '!=' null 2024-09-16T16:09:36.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:36.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:36.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:36.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:36.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:36.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:36.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:36.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:36.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:36.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:36.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:36.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:36.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:36.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:36.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:37.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:37.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:37.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:37.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:37.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:37.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:37.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:37.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:37.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:38.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713463 2024-09-16T16:09:38.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713463 '!=' null 2024-09-16T16:09:38.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:38.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:38.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:38.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:38.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:38.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:38.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:38.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:38.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:38.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:38.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:38.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:38.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:38.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:38.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:39.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:39.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:39.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:39.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:39.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:39.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:39.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:39.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:39.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828376 2024-09-16T16:09:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828376 '!=' null 2024-09-16T16:09:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:40.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:40.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:40.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:40.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:40.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:40.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:40.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:40.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:40.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:40.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:40.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:40.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:41.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:41.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:41.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:41.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:41.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:41.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:41.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:41.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:41.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828328 2024-09-16T16:09:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828328 '!=' null 2024-09-16T16:09:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:41.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:41.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:41.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:41.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:41.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:41.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:41.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:41.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:41.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:42.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:42.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:42.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:42.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:42.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:42.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:42.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:42.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:42.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:42.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:42.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:42.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:42.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:43.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828014 2024-09-16T16:09:43.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828014 '!=' null 2024-09-16T16:09:43.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:43.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:43.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:43.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:43.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:43.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:43.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:43.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:43.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:43.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:43.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:44.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:44.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:44.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:44.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:44.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:44.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:44.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:44.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:44.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:44.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:44.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:44.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:44.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:45.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954476 2024-09-16T16:09:45.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954476 '!=' null 2024-09-16T16:09:45.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:45.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:45.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:45.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:45.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:45.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:45.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:45.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:45.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:45.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:45.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:46.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:46.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:46.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:46.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:46.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:46.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:46.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:46.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:46.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:46.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:46.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:46.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:46.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:47.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839298 2024-09-16T16:09:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839298 '!=' null 2024-09-16T16:09:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:47.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:47.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:47.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:47.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:47.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:47.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:47.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:47.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:47.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:48.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:48.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:48.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:48.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:48.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:48.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:48.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:48.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:48.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:48.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:48.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:48.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:48.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:49.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080495 2024-09-16T16:09:49.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080495 '!=' null 2024-09-16T16:09:49.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:49.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:49.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:49.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:49.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:49.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:49.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:49.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:49.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:49.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:49.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:49.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:49.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:49.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:49.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:50.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:50.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:50.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:50.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:50.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:50.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:50.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:50.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:50.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:51.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080377 2024-09-16T16:09:51.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080377 '!=' null 2024-09-16T16:09:51.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:51.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:51.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:51.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:51.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:51.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:51.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:51.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:51.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:51.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:51.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:52.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:52.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:52.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:52.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:52.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:52.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:52.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:52.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:52.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:52.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:52.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:52.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:52.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:53.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080377 2024-09-16T16:09:53.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080377 '!=' null 2024-09-16T16:09:53.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:53.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:53.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:53.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:53.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:53.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:53.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:53.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:53.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:53.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:53.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:53.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:53.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:53.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:53.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:54.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:54.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:54.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:54.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:54.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:54.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:54.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:54.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:54.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:55.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27953655 2024-09-16T16:09:55.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27953655 '!=' null 2024-09-16T16:09:55.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:55.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:55.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:55.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:55.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:55.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:55.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:55.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:55.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:55.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:55.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:55.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:55.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:55.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:55.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:56.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:56.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:56.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:56.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:56.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:56.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:56.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:57.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827226 2024-09-16T16:09:57.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827226 '!=' null 2024-09-16T16:09:57.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:57.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:57.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:57.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:57.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:57.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:57.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:57.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:57.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:57.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:57.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:57.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:57.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:57.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:57.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:09:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:09:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:09:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:09:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:09:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:09:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:09:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:09:58.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:09:58.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:09:58.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585904 2024-09-16T16:09:58.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585904 '!=' null 2024-09-16T16:09:58.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:09:58.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:09:58.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:09:58.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:09:58.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:09:58.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:09:58.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:09:58.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:09:58.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:09:58.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:09:58.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:09:59.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:09:59.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:09:59.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:09:59.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:00.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:00.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:00.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:00.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:00.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:00.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:00.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:00.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:00.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701461 2024-09-16T16:10:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701461 '!=' null 2024-09-16T16:10:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:00.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:00.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:00.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:00.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:00.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:00.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:00.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:00.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:00.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:01.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:01.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:01.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:01.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:02.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:02.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:02.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:02.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:02.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:02.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:02.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:02.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:02.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:02.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701550 2024-09-16T16:10:02.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701550 '!=' null 2024-09-16T16:10:02.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:02.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:02.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:02.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:02.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:02.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:02.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:02.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:02.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:02.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:02.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:03.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:03.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:03.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:03.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:03.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:03.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:03.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:03.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:03.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:03.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:03.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:03.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:03.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:04.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23587075 2024-09-16T16:10:04.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23587075 '!=' null 2024-09-16T16:10:04.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:04.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:04.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:04.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:04.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:04.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:04.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:04.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:04.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:04.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:04.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:05.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:05.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:05.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:05.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:06.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:06.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:06.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:06.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:06.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:06.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:06.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:06.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:06.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:06.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28829252 2024-09-16T16:10:06.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28829252 '!=' null 2024-09-16T16:10:06.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:06.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:06.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:06.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:06.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:06.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:06.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:06.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:06.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:06.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:06.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:07.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:07.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:07.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:07.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:07.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:07.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:07.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:07.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:07.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:07.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:07.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:07.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:07.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:08.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22714187 2024-09-16T16:10:08.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22714187 '!=' null 2024-09-16T16:10:08.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:08.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:08.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:08.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:08.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:08.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:08.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:08.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:08.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:08.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:08.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:09.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:09.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:09.013 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:09.013 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:09.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:09.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:09.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:09.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:09.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:09.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:09.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:09.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:09.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:10.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27956386 2024-09-16T16:10:10.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27956386 '!=' null 2024-09-16T16:10:10.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:10.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:10.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:10.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:10.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:10.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:10.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:10.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:10.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:10.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:10.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:11.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:11.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:11.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:11.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:11.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:11.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:11.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:11.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:11.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:11.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:11.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:11.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:11.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:12.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27082859 2024-09-16T16:10:12.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27082859 '!=' null 2024-09-16T16:10:12.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:12.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:12.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:12.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:12.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:12.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:12.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:12.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:12.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:12.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:12.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:12.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:12.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:12.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:12.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:13.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:13.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:13.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:13.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:13.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:13.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:13.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:13.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:13.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:14.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21841110 2024-09-16T16:10:14.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21841110 '!=' null 2024-09-16T16:10:14.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:14.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:14.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:14.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:14.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:14.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:14.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:14.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:14.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:14.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:14.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:14.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:14.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:14.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:14.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:15.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:15.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:15.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:15.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:15.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:15.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:15.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:15.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:15.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:16.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34071704 2024-09-16T16:10:16.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34071704 '!=' null 2024-09-16T16:10:16.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:16.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:16.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:16.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:16.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:16.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:16.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:16.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:16.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:16.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:16.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:16.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:16.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:16.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:16.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:17.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:17.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:17.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:17.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:17.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:17.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:17.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:17.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:17.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:18.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713857 2024-09-16T16:10:18.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713857 '!=' null 2024-09-16T16:10:18.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:18.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:18.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:18.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:18.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:18.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:18.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:18.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:18.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:18.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:18.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:18.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:18.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:18.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:18.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:19.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:19.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:19.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:19.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:19.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:19.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:19.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:19.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:19.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:20.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828515 2024-09-16T16:10:20.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828515 '!=' null 2024-09-16T16:10:20.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:20.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:20.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:20.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:20.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:20.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:20.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:20.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:20.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:20.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:20.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:20.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:20.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:20.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:20.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:21.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:21.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:21.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:21.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:21.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:21.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:21.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:21.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:21.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:21.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828154 2024-09-16T16:10:21.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828154 '!=' null 2024-09-16T16:10:21.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:21.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:21.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:21.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:21.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:21.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:21.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:21.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:21.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:21.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:21.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:22.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:22.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:22.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:22.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:23.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:23.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:23.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:23.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:23.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:23.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:23.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:23.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:23.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:23.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586150 2024-09-16T16:10:23.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586150 '!=' null 2024-09-16T16:10:23.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:23.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:23.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:23.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:23.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:23.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:23.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:23.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:23.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:23.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:23.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:24.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:24.383 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:24.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:24.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:24.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:24.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:24.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:24.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:24.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:24.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:24.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:24.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:24.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:25.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34942129 2024-09-16T16:10:25.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34942129 '!=' null 2024-09-16T16:10:25.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:25.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:25.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:25.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:25.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:25.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:25.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:25.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:25.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:25.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:25.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:26.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:26.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:26.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:26.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:26.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:26.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:26.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:26.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:27.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34942129 2024-09-16T16:10:27.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34942129 '!=' null 2024-09-16T16:10:27.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:27.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:27.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:27.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:27.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:27.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:27.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:27.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:27.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:27.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:27.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:28.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:28.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:28.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:28.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:29.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:29.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:29.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:29.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:29.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:29.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:29.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:29.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:29.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:30.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827561 2024-09-16T16:10:30.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827561 '!=' null 2024-09-16T16:10:30.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:30.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:30.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:30.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:30.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:30.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:30.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:30.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:30.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:30.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:30.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:30.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:30.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:30.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:30.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:31.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:31.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:31.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:31.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:31.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:31.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:31.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:31.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:31.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:32.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954347 2024-09-16T16:10:32.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954347 '!=' null 2024-09-16T16:10:32.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:32.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:32.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:32.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:32.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:32.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:32.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:32.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:32.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:32.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:32.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:32.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:32.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:32.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:32.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:33.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:33.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:33.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:33.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:33.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:33.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:33.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:33.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:33.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712944 2024-09-16T16:10:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712944 '!=' null 2024-09-16T16:10:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:33.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:33.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:33.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:33.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:33.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:33.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:33.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:33.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:33.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:34.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:34.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:34.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:34.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:35.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:35.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:35.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:35.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34069906 2024-09-16T16:10:35.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34069906 '!=' null 2024-09-16T16:10:35.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:35.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:35.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:35.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:35.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:35.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:35.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:35.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:35.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:35.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:35.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:36.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:36.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:36.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:36.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:36.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:36.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:36.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:36.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:36.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:36.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:36.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:36.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:36.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:37.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713279 2024-09-16T16:10:37.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713279 '!=' null 2024-09-16T16:10:37.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:37.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:37.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:37.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:37.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:37.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:37.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:37.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:37.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:37.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:38.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:38.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:38.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:38.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:38.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:38.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:38.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:38.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:38.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:38.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:38.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:38.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:38.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:39.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828257 2024-09-16T16:10:39.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828257 '!=' null 2024-09-16T16:10:39.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:39.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:39.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:39.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:39.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:39.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:39.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:39.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:39.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:39.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:39.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:40.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:40.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:40.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:40.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:40.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:40.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:40.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:40.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:40.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:40.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:40.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:40.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:40.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:41.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954297 2024-09-16T16:10:41.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954297 '!=' null 2024-09-16T16:10:41.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:41.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:41.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:41.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:41.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:41.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:41.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:41.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:41.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:41.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:41.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:41.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:41.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:41.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:41.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:42.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:42.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:42.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:42.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:42.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:42.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:42.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:42.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:42.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:43.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954297 2024-09-16T16:10:43.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954297 '!=' null 2024-09-16T16:10:43.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:43.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:43.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:43.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:43.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:43.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:43.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:43.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:43.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:43.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:43.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:43.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:10:43.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:43.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:43.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-16T16:10:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:10:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:10:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:10:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:10:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:10:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:10:44.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:10:44.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:10:44.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17034218 2024-09-16T16:10:44.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17034218 '!=' null 2024-09-16T16:10:44.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:10:44.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:10:44.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:10:44.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:10:44.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:10:44.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:10:44.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:10:44.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:10:44.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:10:44.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:10:44.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:10:45.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:10:45.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:10:45.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:10:45.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:10:45.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-16T16:10:45.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:10:45.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:10:45.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:525: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2024-09-16T16:10:45.716 INFO:tasks.workunit.client.0.smithi039.stdout:TEST PASSED 2024-09-16T16:10:45.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:529: TEST_recovery_pool_priority: echo TEST PASSED 2024-09-16T16:10:45.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:532: TEST_recovery_pool_priority: delete_pool test1 2024-09-16T16:10:45.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-09-16T16:10:45.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-09-16T16:10:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test1' does not exist 2024-09-16T16:10:46.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:533: TEST_recovery_pool_priority: delete_pool test2 2024-09-16T16:10:46.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-09-16T16:10:46.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-09-16T16:10:46.806 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test2' does not exist 2024-09-16T16:10:46.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:534: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2024-09-16T16:10:46.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:10:46.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:10:46.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:10:46.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:10:46.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:10:52.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:10:52.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:535: TEST_recovery_pool_priority: return 0 2024-09-16T16:10:52.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-09-16T16:10:52.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-16T16:10:52.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:10:52.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-16T16:10:52.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:10:52.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:10:52.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:10:52.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:10:52.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:10:52.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:10:52.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:10:52.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:10:52.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:10:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:10:52.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:10:52.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:10:52.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:10:52.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:10:52.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:10:52.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:10:52.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:10:52.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:10:52.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-16T16:10:52.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:10:52.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:10:52.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:10:52.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.203397 2024-09-16T16:10:52.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:10:52.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:10:52.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:10:52.206 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:10:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:10:52.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:10:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:10:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:10:52.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:10:52.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:10:52.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:10:52.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:10:52.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:10:52.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:10:52.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:10:52.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-16T16:10:52.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:10:52.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:10:52.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:10:52.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.203397 2024-09-16T16:10:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:10:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:10:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-09-16T16:10:52.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:10:52.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:10:52.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:10:52.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.203397 2024-09-16T16:10:52.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:10:52.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T16:10:52.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:10:52.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-09-16T16:10:52.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_priority td/osd-recovery-prio 2024-09-16T16:10:52.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local dir=td/osd-recovery-prio 2024-09-16T16:10:52.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:48: TEST_recovery_priority: local pools=10 2024-09-16T16:10:52.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: local OSDS=5 2024-09-16T16:10:52.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: local max_tries=10 2024-09-16T16:10:52.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:52: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2024-09-16T16:10:52.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-09-16T16:10:52.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:10:52.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:10:52.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:10:52.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-09-16T16:10:52.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-09-16T16:10:52.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:10:52.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:10:52.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:10:52.617 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:10:52.617 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:10:52.617 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:10:52.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:10:52.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:10:52.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:10:52.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:10:52.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:10:52.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:10:52.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:10:52.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:10:52.836 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:10:52.836 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:10:52.836 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:10:52.837 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:10:52.837 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:10:52.837 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:10:52.837 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-mon.a.asok 2024-09-16T16:10:52.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:10:52.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.203397/ceph-mon.a.asok config get fsid 2024-09-16T16:10:52.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:10:52.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:10:52.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:10:52.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:10:52.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:10:52.958 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:10:52.958 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:10:52.958 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:10:52.958 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:10:52.958 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:10:52.958 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:10:52.958 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-mon.a.asok 2024-09-16T16:10:52.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:10:52.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.203397/ceph-mon.a.asok config get mon_host 2024-09-16T16:10:53.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2024-09-16T16:10:53.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-09-16T16:10:53.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:10:53.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:10:53.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:10:53.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-09-16T16:10:53.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:10:53.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:10:53.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:10:53.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:10:53.987 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:10:53.987 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:10:53.987 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:10:53.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:10:53.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:10:53.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:10:54.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:54: TEST_recovery_priority: export CEPH_ARGS 2024-09-16T16:10:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: expr 5 - 1 2024-09-16T16:10:54.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: seq 0 4 2024-09-16T16:10:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T16:10:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2024-09-16T16:10:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-16T16:10:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:10:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:10:54.055 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:10:54.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:10:54.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-09-16T16:10:54.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:10:54.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d637accb-056f-47db-8c59-ca3ac8cb7bd0 2024-09-16T16:10:54.057 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 d637accb-056f-47db-8c59-ca3ac8cb7bd0 2024-09-16T16:10:54.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d637accb-056f-47db-8c59-ca3ac8cb7bd0' 2024-09-16T16:10:54.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:10:54.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOWOhmnLqtBBAAOzYIGfcHv+EMLpuK/8j5fg== 2024-09-16T16:10:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOWOhmnLqtBBAAOzYIGfcHv+EMLpuK/8j5fg=="}' 2024-09-16T16:10:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d637accb-056f-47db-8c59-ca3ac8cb7bd0 -i td/osd-recovery-prio/0/new.json 2024-09-16T16:10:54.406 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:10:54.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-09-16T16:10:54.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOWOhmnLqtBBAAOzYIGfcHv+EMLpuK/8j5fg== --osd-uuid d637accb-056f-47db-8c59-ca3ac8cb7bd0 2024-09-16T16:10:54.459 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:10:54.455+0000 7fe8ef41d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:10:54.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:10:54.455+0000 7fe8ef41d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:10:54.463 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:10:54.459+0000 7fe8ef41d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:10:54.463 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:10:54.459+0000 7fe8ef41d7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-16T16:10:54.463 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:10:54.459+0000 7fe8ef41d7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-16T16:10:54.463 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:10:54.459+0000 7fe8ef41d7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-16T16:10:54.506 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:10:54.503+0000 7fe8ef41d7c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-09-16T16:11:00.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-09-16T16:11:00.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:11:00.720 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:11:00.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:11:00.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:11:01.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:11:01.293 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:11:01.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:11:01.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:11:01.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:11:01.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:11:01.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:01.315+0000 7ffab51a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:01.338 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:01.335+0000 7ffab51a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:01.340 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:01.335+0000 7ffab51a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:01.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:11:01.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:11:01.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:11:01.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:11:01.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:11:01.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:01.704 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:11:01.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:11:01.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:01.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:11:01.883 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:01.879+0000 7ffab51a77c0 -1 Falling back to public interface 2024-09-16T16:11:02.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:03.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:03.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:03.119 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:11:03.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:11:03.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:03.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:11:03.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:04.466 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:04.459+0000 7ffab51a77c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:11:04.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:04.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:04.560 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:11:04.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:11:04.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:04.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:11:05.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:06.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:06.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:06.105 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:11:06.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:11:06.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:06.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:11:06.542 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3959050823,v1:127.0.0.1:6803/3959050823] [v2:127.0.0.1:6804/3959050823,v1:127.0.0.1:6805/3959050823] exists,up d637accb-056f-47db-8c59-ca3ac8cb7bd0 2024-09-16T16:11:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:11:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:11:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:11:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T16:11:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2024-09-16T16:11:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-16T16:11:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:11:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:11:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:11:06.547 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:11:06.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:11:06.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:11:06.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-09-16T16:11:06.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:11:06.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eae66c01-fda1-426a-b4a6-7ec9866f651e 2024-09-16T16:11:06.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 eae66c01-fda1-426a-b4a6-7ec9866f651e' 2024-09-16T16:11:06.550 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 eae66c01-fda1-426a-b4a6-7ec9866f651e 2024-09-16T16:11:06.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:11:06.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCaWOhmNxTKIRAAVvXe4En/9DscyzjEJipayQ== 2024-09-16T16:11:06.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCaWOhmNxTKIRAAVvXe4En/9DscyzjEJipayQ=="}' 2024-09-16T16:11:06.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eae66c01-fda1-426a-b4a6-7ec9866f651e -i td/osd-recovery-prio/1/new.json 2024-09-16T16:11:07.108 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:11:07.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-09-16T16:11:07.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCaWOhmNxTKIRAAVvXe4En/9DscyzjEJipayQ== --osd-uuid eae66c01-fda1-426a-b4a6-7ec9866f651e 2024-09-16T16:11:07.159 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:07.159+0000 7f59a561d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:07.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:07.159+0000 7f59a561d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:07.162 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:07.163+0000 7f59a561d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:07.162 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:07.163+0000 7f59a561d7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-16T16:11:07.163 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:07.163+0000 7f59a561d7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-16T16:11:07.163 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:07.163+0000 7f59a561d7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-16T16:11:07.265 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:07.263+0000 7f59a561d7c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-09-16T16:11:13.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-09-16T16:11:13.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:11:13.094 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:11:13.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:11:13.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:11:13.672 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:11:13.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:11:13.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:11:13.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:11:13.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:11:13.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:11:13.708 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:13.707+0000 7f68ad3f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:13.719 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:13.719+0000 7f68ad3f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:13.720 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:13.719+0000 7f68ad3f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:14.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:11:14.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:11:14.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:11:14.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:11:14.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:11:14.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:14.088 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:11:14.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:11:14.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:11:14.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:14.279 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:14.279+0000 7f68ad3f17c0 -1 Falling back to public interface 2024-09-16T16:11:14.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:15.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:15.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:15.514 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:11:15.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:11:15.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:15.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:11:15.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:16.861 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:16.858+0000 7f68ad3f17c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:11:16.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:16.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:16.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:11:16.999 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:11:16.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:16.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:11:17.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:18.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:18.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:18.639 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:11:18.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:11:18.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:18.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:11:19.073 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3141030179,v1:127.0.0.1:6811/3141030179] [v2:127.0.0.1:6812/3141030179,v1:127.0.0.1:6813/3141030179] exists,up eae66c01-fda1-426a-b4a6-7ec9866f651e 2024-09-16T16:11:19.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:11:19.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:11:19.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:11:19.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T16:11:19.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2024-09-16T16:11:19.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-16T16:11:19.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:11:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T16:11:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:11:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/2 2024-09-16T16:11:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-16T16:11:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:11:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:11:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:11:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2024-09-16T16:11:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2024-09-16T16:11:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:11:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:11:19.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-16T16:11:19.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-16T16:11:19.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:11:19.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:11:19.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:11:19.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:11:19.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:11:19.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:11:19.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/2 2024-09-16T16:11:19.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:11:19.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5e82ed78-1a75-443d-8f78-f35339e7091c 2024-09-16T16:11:19.080 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 5e82ed78-1a75-443d-8f78-f35339e7091c 2024-09-16T16:11:19.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5e82ed78-1a75-443d-8f78-f35339e7091c' 2024-09-16T16:11:19.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:11:19.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnWOhmwgoxBhAA2Dwlaeyh5AMIcHCIx5ohcg== 2024-09-16T16:11:19.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnWOhmwgoxBhAA2Dwlaeyh5AMIcHCIx5ohcg=="}' 2024-09-16T16:11:19.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5e82ed78-1a75-443d-8f78-f35339e7091c -i td/osd-recovery-prio/2/new.json 2024-09-16T16:11:19.612 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:11:19.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/2/new.json 2024-09-16T16:11:19.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnWOhmwgoxBhAA2Dwlaeyh5AMIcHCIx5ohcg== --osd-uuid 5e82ed78-1a75-443d-8f78-f35339e7091c 2024-09-16T16:11:19.658 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:19.654+0000 7feadf5497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:19.660 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:19.658+0000 7feadf5497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:19.661 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:19.658+0000 7feadf5497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:19.662 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:19.662+0000 7feadf5497c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-09-16T16:11:19.662 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:19.662+0000 7feadf5497c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-09-16T16:11:19.662 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:19.662+0000 7feadf5497c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-09-16T16:11:19.703 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:19.702+0000 7feadf5497c0 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2024-09-16T16:11:26.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2024-09-16T16:11:26.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:11:26.196 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T16:11:26.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T16:11:26.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:11:26.775 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T16:11:26.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T16:11:26.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:11:26.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:11:26.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:11:26.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:11:26.811 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:26.810+0000 7f91cf0b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:26.813 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:26.810+0000 7f91cf0b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:26.814 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:26.814+0000 7f91cf0b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:27.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T16:11:27.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:11:27.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T16:11:27.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:11:27.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:11:27.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:27.261 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:11:27.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:11:27.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:27.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:11:27.395 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:27.394+0000 7f91cf0b57c0 -1 Falling back to public interface 2024-09-16T16:11:27.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:28.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:28.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:28.728 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:11:28.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:11:28.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:28.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:11:29.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:29.938 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:29.938+0000 7f91cf0b57c0 -1 osd.2 0 log_to_monitors true 2024-09-16T16:11:30.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:30.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:30.165 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:11:30.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:11:30.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:30.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:11:30.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:31.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:31.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:31.680 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:11:31.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:11:31.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:31.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:11:32.148 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/593778316,v1:127.0.0.1:6819/593778316] [v2:127.0.0.1:6820/593778316,v1:127.0.0.1:6821/593778316] exists,up 5e82ed78-1a75-443d-8f78-f35339e7091c 2024-09-16T16:11:32.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:11:32.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:11:32.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:11:32.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T16:11:32.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2024-09-16T16:11:32.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-16T16:11:32.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:11:32.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T16:11:32.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:11:32.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/3 2024-09-16T16:11:32.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-16T16:11:32.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:11:32.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:11:32.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:11:32.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2024-09-16T16:11:32.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2024-09-16T16:11:32.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:11:32.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:11:32.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-16T16:11:32.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:11:32.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:11:32.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:11:32.156 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:11:32.156 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:11:32.156 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:11:32.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:11:32.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:11:32.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:11:32.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:11:32.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:11:32.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-16T16:11:32.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-16T16:11:32.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:11:32.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:11:32.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:11:32.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:11:32.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:11:32.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:11:32.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/3 2024-09-16T16:11:32.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:11:32.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f29f2333-d914-44f5-9073-df527e81d1b1 2024-09-16T16:11:32.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 f29f2333-d914-44f5-9073-df527e81d1b1' 2024-09-16T16:11:32.158 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 f29f2333-d914-44f5-9073-df527e81d1b1 2024-09-16T16:11:32.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:11:32.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0WOhmI51LChAACdjPawXOtHm3BW6aZEXd1A== 2024-09-16T16:11:32.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0WOhmI51LChAACdjPawXOtHm3BW6aZEXd1A=="}' 2024-09-16T16:11:32.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f29f2333-d914-44f5-9073-df527e81d1b1 -i td/osd-recovery-prio/3/new.json 2024-09-16T16:11:32.724 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:11:32.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/3/new.json 2024-09-16T16:11:32.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0WOhmI51LChAACdjPawXOtHm3BW6aZEXd1A== --osd-uuid f29f2333-d914-44f5-9073-df527e81d1b1 2024-09-16T16:11:32.784 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:32.782+0000 7f53af6ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:32.805 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:32.802+0000 7f53af6ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:32.807 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:32.806+0000 7f53af6ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:32.806+0000 7f53af6ab7c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-09-16T16:11:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:32.806+0000 7f53af6ab7c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-09-16T16:11:32.808 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:32.806+0000 7f53af6ab7c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-09-16T16:11:32.897 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:32.894+0000 7f53af6ab7c0 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2024-09-16T16:11:38.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2024-09-16T16:11:38.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:11:38.928 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T16:11:38.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T16:11:38.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:11:39.489 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T16:11:39.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T16:11:39.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:11:39.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:11:39.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:11:39.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:11:39.525 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:39.522+0000 7f6d198977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:39.553 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:39.550+0000 7f6d198977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:39.554 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:39.554+0000 7f6d198977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:39.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T16:11:39.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:11:39.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T16:11:39.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:11:39.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:11:39.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:39.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:11:39.978 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:11:39.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:39.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:11:40.115 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:40.114+0000 7f6d198977c0 -1 Falling back to public interface 2024-09-16T16:11:40.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:41.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:41.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:41.409 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:11:41.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:11:41.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:41.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:11:41.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:42.689 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:42.686+0000 7f6d198977c0 -1 osd.3 0 log_to_monitors true 2024-09-16T16:11:42.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:42.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:42.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:11:42.832 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:11:42.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:42.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:11:43.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:44.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:44.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:44.354 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:11:44.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:11:44.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:44.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:11:44.836 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1545024806,v1:127.0.0.1:6827/1545024806] [v2:127.0.0.1:6828/1545024806,v1:127.0.0.1:6829/1545024806] exists,up f29f2333-d914-44f5-9073-df527e81d1b1 2024-09-16T16:11:44.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:11:44.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:11:44.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:11:44.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T16:11:44.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2024-09-16T16:11:44.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-16T16:11:44.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/4 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-16T16:11:44.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-16T16:11:44.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:11:44.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:11:44.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:11:44.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:11:44.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:11:44.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:11:44.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/4 2024-09-16T16:11:44.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:11:44.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c677276b-25c5-4036-b358-43e7fffe41ed 2024-09-16T16:11:44.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 c677276b-25c5-4036-b358-43e7fffe41ed' 2024-09-16T16:11:44.844 INFO:tasks.workunit.client.0.smithi039.stdout:add osd4 c677276b-25c5-4036-b358-43e7fffe41ed 2024-09-16T16:11:44.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:11:44.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAWOhmErBpMxAANZ1iTiTju+9wxk/nL9NMXg== 2024-09-16T16:11:44.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAWOhmErBpMxAANZ1iTiTju+9wxk/nL9NMXg=="}' 2024-09-16T16:11:44.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c677276b-25c5-4036-b358-43e7fffe41ed -i td/osd-recovery-prio/4/new.json 2024-09-16T16:11:45.485 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:11:45.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/4/new.json 2024-09-16T16:11:45.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDAWOhmErBpMxAANZ1iTiTju+9wxk/nL9NMXg== --osd-uuid c677276b-25c5-4036-b358-43e7fffe41ed 2024-09-16T16:11:45.539 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:45.538+0000 7f08e825e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:45.565 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:45.562+0000 7f08e825e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:45.566 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:45.566+0000 7f08e825e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:45.567 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:45.566+0000 7f08e825e7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-09-16T16:11:45.567 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:45.566+0000 7f08e825e7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-09-16T16:11:45.567 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:45.566+0000 7f08e825e7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-09-16T16:11:45.640 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:45.638+0000 7f08e825e7c0 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2024-09-16T16:11:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2024-09-16T16:11:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:11:51.900 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd4 key to auth repository 2024-09-16T16:11:51.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T16:11:51.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:11:52.467 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.4 2024-09-16T16:11:52.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T16:11:52.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1049f195-64a5-4d4a-b7bd-58c0641087e5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.203397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:11:52.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:11:52.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:11:52.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:11:52.502 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:52.502+0000 7fd9308767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:52.514 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:52.514+0000 7fd9308767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:52.515 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:52.514+0000 7fd9308767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:11:52.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T16:11:52.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:11:52.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T16:11:52.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:11:52.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:11:52.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:52.891 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:11:52.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:11:52.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:52.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:11:53.083 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:53.082+0000 7fd9308767c0 -1 Falling back to public interface 2024-09-16T16:11:53.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:54.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:54.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:54.342 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:11:54.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:11:54.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:54.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:11:54.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:55.756 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:11:55.754+0000 7fd9308767c0 -1 osd.4 0 log_to_monitors true 2024-09-16T16:11:55.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:55.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:55.818 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:11:55.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:11:55.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:55.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:11:56.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:11:57.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:11:57.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:11:57.325 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:11:57.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:11:57.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:11:57.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:11:57.759 INFO:tasks.workunit.client.0.smithi039.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2686817302,v1:127.0.0.1:6835/2686817302] [v2:127.0.0.1:6836/2686817302,v1:127.0.0.1:6837/2686817302] exists,up c677276b-25c5-4036-b358-43e7fffe41ed 2024-09-16T16:11:57.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:11:57.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:11:57.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:11:57.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: seq 1 10 2024-09-16T16:11:57.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:11:57.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test1 1 1 2024-09-16T16:11:57.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-09-16T16:11:58.363 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test1' already exists 2024-09-16T16:11:58.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:11:59.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test1 size 2 2024-09-16T16:12:00.515 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-09-16T16:12:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test2 1 1 2024-09-16T16:12:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-09-16T16:12:01.433 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test2' already exists 2024-09-16T16:12:01.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:12:02.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test2 size 2 2024-09-16T16:12:03.466 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 2 size to 2 2024-09-16T16:12:03.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:03.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test3 1 1 2024-09-16T16:12:03.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-09-16T16:12:04.132 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test3' already exists 2024-09-16T16:12:04.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:12:05.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-09-16T16:12:06.082 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 3 size to 2 2024-09-16T16:12:06.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:06.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test4 1 1 2024-09-16T16:12:06.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2024-09-16T16:12:07.157 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test4' already exists 2024-09-16T16:12:07.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:12:08.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test4 size 2 2024-09-16T16:12:09.032 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 4 size to 2 2024-09-16T16:12:09.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:09.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test5 1 1 2024-09-16T16:12:09.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2024-09-16T16:12:10.072 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test5' already exists 2024-09-16T16:12:10.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:12:11.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test5 size 2 2024-09-16T16:12:12.061 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 5 size to 2 2024-09-16T16:12:12.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:12.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test6 1 1 2024-09-16T16:12:12.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2024-09-16T16:12:13.000 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test6' already exists 2024-09-16T16:12:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:12:14.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test6 size 2 2024-09-16T16:12:14.881 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 6 size to 2 2024-09-16T16:12:14.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:14.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test7 1 1 2024-09-16T16:12:14.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2024-09-16T16:12:15.988 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test7' already exists 2024-09-16T16:12:16.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:12:17.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-09-16T16:12:18.065 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 7 size to 2 2024-09-16T16:12:18.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:18.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test8 1 1 2024-09-16T16:12:18.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2024-09-16T16:12:19.098 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test8' already exists 2024-09-16T16:12:19.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:12:20.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test8 size 2 2024-09-16T16:12:20.975 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 8 size to 2 2024-09-16T16:12:20.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:20.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test9 1 1 2024-09-16T16:12:20.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2024-09-16T16:12:21.813 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test9' already exists 2024-09-16T16:12:21.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:12:22.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test9 size 2 2024-09-16T16:12:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 9 size to 2 2024-09-16T16:12:23.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:23.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test10 1 1 2024-09-16T16:12:23.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2024-09-16T16:12:24.436 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test10' already exists 2024-09-16T16:12:24.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:12:25.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test10 size 2 2024-09-16T16:12:26.351 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 10 size to 2 2024-09-16T16:12:26.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:66: TEST_recovery_priority: sleep 5 2024-09-16T16:12:31.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:68: TEST_recovery_priority: wait_for_clean 2024-09-16T16:12:31.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:12:31.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:12:31.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:12:31.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:12:31.375 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:12:31.375 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:12:31.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:12:31.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:12:31.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:12:31.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:12:31.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:12:31.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:12:31.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:12:31.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:12:31.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:12:31.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:12:31.965 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:12:31.965 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:12:31.965 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:12:31.965 INFO:tasks.workunit.client.0.smithi039.stderr:4' 2024-09-16T16:12:31.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:12:31.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:12:31.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:12:32.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836500 2024-09-16T16:12:32.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836500 2024-09-16T16:12:32.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500' 2024-09-16T16:12:32.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:12:32.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:12:32.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672977 2024-09-16T16:12:32.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672977 2024-09-16T16:12:32.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500 1-42949672977' 2024-09-16T16:12:32.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:12:32.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:12:32.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509455 2024-09-16T16:12:32.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509455 2024-09-16T16:12:32.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455' 2024-09-16T16:12:32.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:12:32.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:12:32.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345932 2024-09-16T16:12:32.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345932 2024-09-16T16:12:32.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455 3-85899345932' 2024-09-16T16:12:32.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:12:32.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:12:32.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182410 2024-09-16T16:12:32.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182410 2024-09-16T16:12:32.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455 3-85899345932 4-107374182410' 2024-09-16T16:12:32.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:12:32.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836500 2024-09-16T16:12:32.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:12:32.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:12:32.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836500 2024-09-16T16:12:32.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:12:32.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836500 2024-09-16T16:12:32.807 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836500 2024-09-16T16:12:32.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-16T16:12:32.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:12:33.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-16T16:12:33.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:12:33.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672977 2024-09-16T16:12:33.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:12:33.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:12:33.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672977 2024-09-16T16:12:33.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:12:33.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672977 2024-09-16T16:12:33.236 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672977 2024-09-16T16:12:33.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-16T16:12:33.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:12:33.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-16T16:12:33.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:12:33.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509455 2024-09-16T16:12:33.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:12:33.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:12:33.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509455 2024-09-16T16:12:33.657 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:12:33.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509455 2024-09-16T16:12:33.658 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509455 2024-09-16T16:12:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-16T16:12:33.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:12:34.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-16T16:12:34.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:12:34.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:12:34.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345932 2024-09-16T16:12:34.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:12:34.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345932 2024-09-16T16:12:34.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:12:34.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345932 2024-09-16T16:12:34.090 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345932 2024-09-16T16:12:34.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-09-16T16:12:34.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:12:34.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345932 -lt 85899345932 2024-09-16T16:12:34.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:12:34.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182410 2024-09-16T16:12:34.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:12:34.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:12:34.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182410 2024-09-16T16:12:34.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:12:34.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182410 2024-09-16T16:12:34.519 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182410 2024-09-16T16:12:34.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2024-09-16T16:12:34.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:12:34.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182410 -lt 107374182410 2024-09-16T16:12:34.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:12:34.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:12:34.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:12:35.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2024-09-16T16:12:35.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:12:35.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:12:35.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:12:35.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:12:35.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:12:35.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:12:35.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:12:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2024-09-16T16:12:35.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:12:35.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:12:35.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:12:36.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2024-09-16T16:12:36.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:12:36.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:12:36.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:70: TEST_recovery_priority: ceph pg dump pgs 2024-09-16T16:12:36.862 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T16:12:36.863 INFO:tasks.workunit.client.0.smithi039.stdout:10.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:12:27.279281+0000 0'0 83:21 [2,1] 2 [2,1] 2 0'0 2024-09-16T16:12:24.250287+0000 0'0 2024-09-16T16:12:24.250287+0000 0 0 periodic scrub scheduled @ 2024-09-17T21:50:48.456017+0000 0 0 2024-09-16T16:12:36.863 INFO:tasks.workunit.client.0.smithi039.stdout:9.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:12:24.765852+0000 0'0 82:25 [1,2] 1 [1,2] 1 0'0 2024-09-16T16:12:21.601454+0000 0'0 2024-09-16T16:12:21.601454+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:49:18.858965+0000 0 0 2024-09-16T16:12:36.863 INFO:tasks.workunit.client.0.smithi039.stdout:8.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:12:21.955340+0000 0'0 82:27 [3,4] 3 [3,4] 3 0'0 2024-09-16T16:12:18.649756+0000 0'0 2024-09-16T16:12:18.649756+0000 0 0 periodic scrub scheduled @ 2024-09-18T02:40:13.637477+0000 0 0 2024-09-16T16:12:36.863 INFO:tasks.workunit.client.0.smithi039.stdout:7.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:12:18.761644+0000 0'0 82:35 [1,4] 1 [1,4] 1 0'0 2024-09-16T16:12:15.456943+0000 0'0 2024-09-16T16:12:15.456943+0000 0 0 periodic scrub scheduled @ 2024-09-17T21:52:43.049069+0000 0 0 2024-09-16T16:12:36.863 INFO:tasks.workunit.client.0.smithi039.stdout:6.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:12:15.645483+0000 0'0 82:33 [4,2] 4 [4,2] 4 0'0 2024-09-16T16:12:12.674144+0000 0'0 2024-09-16T16:12:12.674144+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:15:44.720764+0000 0 0 2024-09-16T16:12:36.863 INFO:tasks.workunit.client.0.smithi039.stdout:5.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:12:12.969006+0000 0'0 82:36 [3,4] 3 [3,4] 3 0'0 2024-09-16T16:12:09.506305+0000 0'0 2024-09-16T16:12:09.506305+0000 0 0 periodic scrub scheduled @ 2024-09-17T19:20:41.313109+0000 0 0 2024-09-16T16:12:36.863 INFO:tasks.workunit.client.0.smithi039.stdout:4.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:12:09.735387+0000 0'0 82:40 [4,0] 4 [4,0] 4 0'0 2024-09-16T16:12:06.615684+0000 0'0 2024-09-16T16:12:06.615684+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:47:16.152291+0000 0 0 2024-09-16T16:12:36.863 INFO:tasks.workunit.client.0.smithi039.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:12:06.886637+0000 0'0 82:51 [1,2] 1 [1,2] 1 0'0 2024-09-16T16:12:03.866975+0000 0'0 2024-09-16T16:12:03.866975+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:14:42.677832+0000 0 0 2024-09-16T16:12:36.863 INFO:tasks.workunit.client.0.smithi039.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:12:04.863132+0000 0'0 82:53 [3,1] 3 [3,1] 3 0'0 2024-09-16T16:12:01.023089+0000 0'0 2024-09-16T16:12:01.023089+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:51:50.286284+0000 0 0 2024-09-16T16:12:36.863 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T16:12:01.512213+0000 0'0 82:52 [1,0] 1 [1,0] 1 0'0 2024-09-16T16:11:58.123419+0000 0'0 2024-09-16T16:11:58.123419+0000 0 0 periodic scrub scheduled @ 2024-09-17T16:58:06.592371+0000 0 0 2024-09-16T16:12:36.864 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:12:36.864 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T16:12:36.864 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:12:36.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local PG1 2024-09-16T16:12:36.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local POOLNUM1 2024-09-16T16:12:36.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:76: TEST_recovery_priority: local pool1 2024-09-16T16:12:36.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local chk_osd1_1 2024-09-16T16:12:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local chk_osd1_2 2024-09-16T16:12:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local PG2 2024-09-16T16:12:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:81: TEST_recovery_priority: local POOLNUM2 2024-09-16T16:12:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local pool2 2024-09-16T16:12:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local chk_osd2 2024-09-16T16:12:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:85: TEST_recovery_priority: local PG3 2024-09-16T16:12:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: local POOLNUM3 2024-09-16T16:12:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:87: TEST_recovery_priority: local pool3 2024-09-16T16:12:36.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: seq 1 10 2024-09-16T16:12:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:36.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-16T16:12:36.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 1.0 --format=json 2024-09-16T16:12:37.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-16T16:12:37.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-09-16T16:12:37.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-16T16:12:37.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-09-16T16:12:37.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z '' ']' 2024-09-16T16:12:37.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: PG1=1.0 2024-09-16T16:12:37.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: POOLNUM1=1 2024-09-16T16:12:37.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: pool1=test1 2024-09-16T16:12:37.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:99: TEST_recovery_priority: chk_osd1_1=1 2024-09-16T16:12:37.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: chk_osd1_2=0 2024-09-16T16:12:37.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:37.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 2.0 --format=json 2024-09-16T16:12:37.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-16T16:12:37.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-16T16:12:37.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-09-16T16:12:37.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-16T16:12:37.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=1 2024-09-16T16:12:37.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-16T16:12:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2024-09-16T16:12:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2024-09-16T16:12:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 3.0 --format=json 2024-09-16T16:12:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-16T16:12:38.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-16T16:12:38.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-09-16T16:12:38.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-16T16:12:38.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-09-16T16:12:38.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-16T16:12:38.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2024-09-16T16:12:38.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: PG2=3.0 2024-09-16T16:12:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: POOLNUM2=3 2024-09-16T16:12:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:105: TEST_recovery_priority: pool2=test3 2024-09-16T16:12:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: chk_osd2=2 2024-09-16T16:12:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 4.0 --format=json 2024-09-16T16:12:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-16T16:12:38.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-16T16:12:38.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-09-16T16:12:38.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-16T16:12:38.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-09-16T16:12:38.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-16T16:12:38.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2024-09-16T16:12:38.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2024-09-16T16:12:38.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:38.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 5.0 --format=json 2024-09-16T16:12:38.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-16T16:12:38.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-16T16:12:38.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-09-16T16:12:38.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-16T16:12:38.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-09-16T16:12:38.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-16T16:12:38.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2024-09-16T16:12:38.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-09-16T16:12:38.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:38.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-16T16:12:38.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 6.0 --format=json 2024-09-16T16:12:39.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-16T16:12:39.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-09-16T16:12:39.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-16T16:12:39.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-09-16T16:12:39.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-16T16:12:39.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2024-09-16T16:12:39.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2024-09-16T16:12:39.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:39.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 7.0 --format=json 2024-09-16T16:12:39.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-16T16:12:39.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-16T16:12:39.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-09-16T16:12:39.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-16T16:12:39.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-09-16T16:12:39.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-16T16:12:39.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2024-09-16T16:12:39.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-09-16T16:12:39.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: PG3=7.0 2024-09-16T16:12:39.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:110: TEST_recovery_priority: POOLNUM3=7 2024-09-16T16:12:39.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:111: TEST_recovery_priority: pool3=test7 2024-09-16T16:12:39.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: break 2024-09-16T16:12:39.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:115: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2024-09-16T16:12:39.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:117: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2024-09-16T16:12:39.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: seq 1 10 2024-09-16T16:12:39.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:39.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2024-09-16T16:12:39.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:39.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2024-09-16T16:12:39.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test2 2024-09-16T16:12:39.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-09-16T16:12:39.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-09-16T16:12:40.540 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test2' does not exist 2024-09-16T16:12:40.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:40.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2024-09-16T16:12:40.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:40.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2024-09-16T16:12:40.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test4 2024-09-16T16:12:40.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2024-09-16T16:12:40.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2024-09-16T16:12:41.366 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test4' does not exist 2024-09-16T16:12:41.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:41.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2024-09-16T16:12:41.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test5 2024-09-16T16:12:41.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2024-09-16T16:12:41.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2024-09-16T16:12:42.002 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test5' does not exist 2024-09-16T16:12:42.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:42.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2024-09-16T16:12:42.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test6 2024-09-16T16:12:42.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2024-09-16T16:12:42.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2024-09-16T16:12:42.759 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test6' does not exist 2024-09-16T16:12:42.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:42.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2024-09-16T16:12:42.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:42.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2024-09-16T16:12:42.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test8 2024-09-16T16:12:42.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2024-09-16T16:12:42.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2024-09-16T16:12:43.420 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test8' does not exist 2024-09-16T16:12:43.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:43.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2024-09-16T16:12:43.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test9 2024-09-16T16:12:43.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2024-09-16T16:12:43.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2024-09-16T16:12:44.097 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test9' does not exist 2024-09-16T16:12:44.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-16T16:12:44.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2024-09-16T16:12:44.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test10 2024-09-16T16:12:44.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2024-09-16T16:12:44.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2024-09-16T16:12:44.713 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test10' does not exist 2024-09-16T16:12:44.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:131: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2024-09-16T16:12:45.769 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 3 size to 1 2024-09-16T16:12:45.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2024-09-16T16:12:47.079 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 7 size to 1 2024-09-16T16:12:47.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: wait_for_clean 2024-09-16T16:12:47.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:12:47.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:12:47.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:12:47.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:12:47.105 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:12:47.105 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:12:47.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:12:47.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:12:47.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:12:47.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:12:47.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:12:47.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:12:47.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:12:47.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:12:47.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:12:47.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:12:47.750 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:12:47.750 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:12:47.750 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:12:47.750 INFO:tasks.workunit.client.0.smithi039.stderr:4' 2024-09-16T16:12:47.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:12:47.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:12:47.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:12:47.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836505 2024-09-16T16:12:47.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836505 2024-09-16T16:12:47.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505' 2024-09-16T16:12:47.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:12:47.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:12:48.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672982 2024-09-16T16:12:48.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672982 2024-09-16T16:12:48.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672982' 2024-09-16T16:12:48.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:12:48.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:12:48.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509460 2024-09-16T16:12:48.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509460 2024-09-16T16:12:48.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672982 2-64424509460' 2024-09-16T16:12:48.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:12:48.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:12:48.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345937 2024-09-16T16:12:48.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345937 2024-09-16T16:12:48.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672982 2-64424509460 3-85899345937' 2024-09-16T16:12:48.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:12:48.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:12:48.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182415 2024-09-16T16:12:48.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182415 2024-09-16T16:12:48.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672982 2-64424509460 3-85899345937 4-107374182415' 2024-09-16T16:12:48.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:12:48.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836505 2024-09-16T16:12:48.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:12:48.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:12:48.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836505 2024-09-16T16:12:48.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:12:48.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836505 2024-09-16T16:12:48.621 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836505 2024-09-16T16:12:48.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-16T16:12:48.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:12:49.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-16T16:12:49.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:12:49.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672982 2024-09-16T16:12:49.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:12:49.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:12:49.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672982 2024-09-16T16:12:49.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:12:49.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672982 2024-09-16T16:12:49.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-16T16:12:49.036 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672982 2024-09-16T16:12:49.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:12:49.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672983 -lt 42949672982 2024-09-16T16:12:49.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:12:49.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509460 2024-09-16T16:12:49.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:12:49.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:12:49.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509460 2024-09-16T16:12:49.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:12:49.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509460 2024-09-16T16:12:49.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-16T16:12:49.502 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509460 2024-09-16T16:12:49.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:12:49.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-16T16:12:49.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:12:49.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345937 2024-09-16T16:12:49.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:12:49.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:12:49.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345937 2024-09-16T16:12:49.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:12:49.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345937 2024-09-16T16:12:49.932 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345937 2024-09-16T16:12:49.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2024-09-16T16:12:49.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:12:50.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345937 -lt 85899345937 2024-09-16T16:12:50.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:12:50.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182415 2024-09-16T16:12:50.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:12:50.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:12:50.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182415 2024-09-16T16:12:50.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:12:50.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182415 2024-09-16T16:12:50.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-09-16T16:12:50.359 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182415 2024-09-16T16:12:50.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:12:50.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182415 -lt 107374182415 2024-09-16T16:12:50.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:12:50.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:12:50.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:12:51.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-09-16T16:12:51.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:12:51.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:12:51.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:12:51.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:12:51.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:12:51.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:12:51.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:12:51.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-16T16:12:51.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:12:51.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:12:51.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:12:52.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-09-16T16:12:52.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:12:52.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:12:52.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:135: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-09-16T16:12:52.441 INFO:tasks.workunit.client.0.smithi039.stderr:10+0 records in 2024-09-16T16:12:52.441 INFO:tasks.workunit.client.0.smithi039.stderr:10+0 records out 2024-09-16T16:12:52.441 INFO:tasks.workunit.client.0.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0390112 s, 269 MB/s 2024-09-16T16:12:52.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: p=1 2024-09-16T16:12:52.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-09-16T16:12:52.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-09-16T16:12:52.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:52.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-09-16T16:12:53.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:53.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-09-16T16:12:53.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:53.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-09-16T16:12:54.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:54.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-09-16T16:12:55.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:55.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-09-16T16:12:55.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:55.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-09-16T16:12:56.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:56.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-09-16T16:12:57.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:57.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-09-16T16:12:57.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:57.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-09-16T16:12:58.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:58.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-09-16T16:12:59.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:59.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-09-16T16:12:59.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:12:59.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-09-16T16:13:00.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:00.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-09-16T16:13:01.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:01.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-09-16T16:13:01.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:01.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-09-16T16:13:02.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:02.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-09-16T16:13:03.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:03.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-09-16T16:13:03.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:03.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-09-16T16:13:04.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:04.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-09-16T16:13:05.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:05.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-09-16T16:13:05.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:05.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-09-16T16:13:06.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:06.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-09-16T16:13:07.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:07.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-09-16T16:13:07.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:07.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-09-16T16:13:08.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:08.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-09-16T16:13:09.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:09.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-09-16T16:13:10.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:10.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-09-16T16:13:10.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:10.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-09-16T16:13:11.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:11.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-09-16T16:13:12.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:12.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-09-16T16:13:12.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:12.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-09-16T16:13:13.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:13.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-09-16T16:13:14.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:14.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-09-16T16:13:14.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:14.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-09-16T16:13:15.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:15.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-09-16T16:13:16.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:16.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-09-16T16:13:16.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:16.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-09-16T16:13:17.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:17.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-09-16T16:13:18.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:18.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-09-16T16:13:18.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:18.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-09-16T16:13:19.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:19.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-09-16T16:13:20.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:20.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-09-16T16:13:21.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:21.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-09-16T16:13:21.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:21.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-09-16T16:13:22.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:22.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-09-16T16:13:23.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:23.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-09-16T16:13:23.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:23.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-09-16T16:13:24.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:24.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-09-16T16:13:25.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:25.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-09-16T16:13:25.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:25.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-09-16T16:13:26.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:26.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-09-16T16:13:27.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:27.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-09-16T16:13:28.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:28.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-09-16T16:13:28.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:28.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-09-16T16:13:29.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:29.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-09-16T16:13:29.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:29.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-09-16T16:13:30.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:30.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-09-16T16:13:31.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:31.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-09-16T16:13:32.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:32.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-09-16T16:13:32.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:32.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-09-16T16:13:33.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:33.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-09-16T16:13:33.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:33.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-09-16T16:13:34.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:34.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-09-16T16:13:35.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:35.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-09-16T16:13:35.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:35.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-09-16T16:13:36.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:36.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-09-16T16:13:37.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:37.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-09-16T16:13:37.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:37.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-09-16T16:13:38.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:38.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-09-16T16:13:39.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:39.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-09-16T16:13:40.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:40.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-09-16T16:13:40.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:40.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-09-16T16:13:41.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:41.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-09-16T16:13:42.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:42.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-09-16T16:13:42.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:42.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-09-16T16:13:43.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:43.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-09-16T16:13:44.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:44.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-09-16T16:13:44.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:44.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-09-16T16:13:45.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:45.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-09-16T16:13:46.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:46.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-09-16T16:13:46.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:46.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-09-16T16:13:47.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:47.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-09-16T16:13:48.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:48.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-09-16T16:13:48.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:48.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-09-16T16:13:49.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:49.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-09-16T16:13:50.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:50.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-09-16T16:13:51.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:51.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-09-16T16:13:51.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:51.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-09-16T16:13:52.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:52.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-09-16T16:13:53.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:53.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-09-16T16:13:53.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:53.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-09-16T16:13:54.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:54.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-09-16T16:13:55.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:55.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-09-16T16:13:55.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:55.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-09-16T16:13:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:56.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-09-16T16:13:57.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:57.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-09-16T16:13:57.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:57.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-09-16T16:13:58.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:58.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-09-16T16:13:59.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:59.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-09-16T16:13:59.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:13:59.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-09-16T16:14:00.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:00.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-09-16T16:14:01.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:01.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-09-16T16:14:01.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:01.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-09-16T16:14:02.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:02.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-09-16T16:14:03.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:03.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-09-16T16:14:04.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:04.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-09-16T16:14:04.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:04.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-09-16T16:14:05.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:05.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-09-16T16:14:06.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:06.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-09-16T16:14:06.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:06.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-09-16T16:14:07.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:07.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-09-16T16:14:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:08.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-09-16T16:14:08.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:08.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-09-16T16:14:09.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:09.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-09-16T16:14:10.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:10.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-09-16T16:14:11.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:11.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-09-16T16:14:11.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:11.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-09-16T16:14:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-09-16T16:14:13.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:13.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-09-16T16:14:13.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:13.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-09-16T16:14:14.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:14.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-09-16T16:14:15.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:15.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-09-16T16:14:15.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:15.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-09-16T16:14:16.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:16.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-09-16T16:14:17.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:17.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-09-16T16:14:18.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:18.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-09-16T16:14:18.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:18.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-09-16T16:14:19.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:19.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-09-16T16:14:20.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:20.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-09-16T16:14:20.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:20.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-09-16T16:14:21.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:21.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-09-16T16:14:22.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:22.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-09-16T16:14:22.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:22.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-09-16T16:14:23.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:23.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-09-16T16:14:24.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:24.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-09-16T16:14:25.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:25.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-09-16T16:14:25.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:25.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-09-16T16:14:26.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:26.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-09-16T16:14:27.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:27.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-09-16T16:14:27.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:27.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-09-16T16:14:28.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:28.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-09-16T16:14:29.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:29.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-09-16T16:14:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:29.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-09-16T16:14:30.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:30.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-09-16T16:14:31.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:31.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-09-16T16:14:32.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:32.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-09-16T16:14:32.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:32.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-09-16T16:14:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-09-16T16:14:34.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:34.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-09-16T16:14:34.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:34.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-09-16T16:14:35.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:35.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-09-16T16:14:36.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:36.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-09-16T16:14:36.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:36.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-09-16T16:14:37.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:37.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-09-16T16:14:37.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:37.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-09-16T16:14:38.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:38.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-09-16T16:14:39.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:39.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-09-16T16:14:39.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:39.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-09-16T16:14:40.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:40.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-09-16T16:14:41.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:41.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-09-16T16:14:42.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:42.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-09-16T16:14:42.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:42.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-09-16T16:14:43.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:43.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-09-16T16:14:44.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:44.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-09-16T16:14:44.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:44.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-09-16T16:14:45.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:45.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-09-16T16:14:46.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:46.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-09-16T16:14:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-09-16T16:14:47.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:47.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-09-16T16:14:48.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:48.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-09-16T16:14:48.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:48.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-09-16T16:14:49.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:49.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-09-16T16:14:50.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:50.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-09-16T16:14:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:50.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-09-16T16:14:51.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:51.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-09-16T16:14:52.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:52.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-09-16T16:14:53.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:53.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-09-16T16:14:53.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:53.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-09-16T16:14:54.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:54.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-09-16T16:14:55.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:55.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-09-16T16:14:55.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:55.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-09-16T16:14:56.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:56.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-09-16T16:14:57.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:57.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-09-16T16:14:58.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:58.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-09-16T16:14:58.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:58.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-09-16T16:14:59.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:14:59.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-09-16T16:15:00.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:00.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-09-16T16:15:00.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:00.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-09-16T16:15:01.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:01.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-09-16T16:15:02.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:02.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-09-16T16:15:02.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:02.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-09-16T16:15:03.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:03.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-09-16T16:15:04.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:04.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-09-16T16:15:05.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:05.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-09-16T16:15:05.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:05.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-09-16T16:15:06.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:06.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-09-16T16:15:07.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:07.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-09-16T16:15:07.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:07.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-09-16T16:15:08.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:08.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-09-16T16:15:09.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:09.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-09-16T16:15:09.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 1 + 1 2024-09-16T16:15:09.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=2 2024-09-16T16:15:09.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-09-16T16:15:09.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-09-16T16:15:09.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:09.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2024-09-16T16:15:10.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:10.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2024-09-16T16:15:10.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:10.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2024-09-16T16:15:11.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:11.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2024-09-16T16:15:11.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:11.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2024-09-16T16:15:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:11.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2024-09-16T16:15:12.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:12.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2024-09-16T16:15:12.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:12.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2024-09-16T16:15:13.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:13.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2024-09-16T16:15:13.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:13.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2024-09-16T16:15:13.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:13.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2024-09-16T16:15:14.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:14.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2024-09-16T16:15:14.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:14.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2024-09-16T16:15:15.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:15.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2024-09-16T16:15:15.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:15.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2024-09-16T16:15:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2024-09-16T16:15:16.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:16.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2024-09-16T16:15:16.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:16.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2024-09-16T16:15:17.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:17.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2024-09-16T16:15:17.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:17.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2024-09-16T16:15:17.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:17.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2024-09-16T16:15:18.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:18.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2024-09-16T16:15:18.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:18.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2024-09-16T16:15:19.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:19.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2024-09-16T16:15:19.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:19.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2024-09-16T16:15:19.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:19.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2024-09-16T16:15:20.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:20.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2024-09-16T16:15:20.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:20.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2024-09-16T16:15:21.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:21.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2024-09-16T16:15:21.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:21.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2024-09-16T16:15:21.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:21.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2024-09-16T16:15:22.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:22.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2024-09-16T16:15:22.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:22.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2024-09-16T16:15:23.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:23.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2024-09-16T16:15:23.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:23.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2024-09-16T16:15:24.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:24.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2024-09-16T16:15:24.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:24.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2024-09-16T16:15:24.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:24.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2024-09-16T16:15:25.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:25.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2024-09-16T16:15:25.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:25.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2024-09-16T16:15:26.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:26.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2024-09-16T16:15:26.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:26.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2024-09-16T16:15:26.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:26.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2024-09-16T16:15:27.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:27.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2024-09-16T16:15:27.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:27.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2024-09-16T16:15:28.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:28.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2024-09-16T16:15:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2024-09-16T16:15:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2024-09-16T16:15:29.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:29.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2024-09-16T16:15:29.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:29.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2024-09-16T16:15:30.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:30.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2024-09-16T16:15:30.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:30.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2024-09-16T16:15:30.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:30.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2024-09-16T16:15:31.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:31.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2024-09-16T16:15:31.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:31.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2024-09-16T16:15:32.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:32.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2024-09-16T16:15:32.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:32.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2024-09-16T16:15:32.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:32.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2024-09-16T16:15:33.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:33.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2024-09-16T16:15:33.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:33.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2024-09-16T16:15:34.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:34.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2024-09-16T16:15:34.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:34.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2024-09-16T16:15:34.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:34.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2024-09-16T16:15:35.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:35.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2024-09-16T16:15:35.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:35.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2024-09-16T16:15:35.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:35.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2024-09-16T16:15:36.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:36.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2024-09-16T16:15:36.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:36.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2024-09-16T16:15:37.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:37.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2024-09-16T16:15:37.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:37.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2024-09-16T16:15:37.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:37.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2024-09-16T16:15:38.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:38.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2024-09-16T16:15:38.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:38.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2024-09-16T16:15:39.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:39.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2024-09-16T16:15:39.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:39.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2024-09-16T16:15:40.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:40.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2024-09-16T16:15:40.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:40.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2024-09-16T16:15:40.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:40.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2024-09-16T16:15:41.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:41.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2024-09-16T16:15:41.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:41.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2024-09-16T16:15:42.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:42.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2024-09-16T16:15:42.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:42.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2024-09-16T16:15:42.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:42.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2024-09-16T16:15:43.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:43.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2024-09-16T16:15:43.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:43.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2024-09-16T16:15:44.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:44.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2024-09-16T16:15:44.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:44.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2024-09-16T16:15:44.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:44.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2024-09-16T16:15:45.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:45.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2024-09-16T16:15:45.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:45.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2024-09-16T16:15:46.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:46.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2024-09-16T16:15:46.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:46.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2024-09-16T16:15:46.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:46.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2024-09-16T16:15:47.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:47.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2024-09-16T16:15:47.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:47.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2024-09-16T16:15:47.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:47.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2024-09-16T16:15:48.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:48.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2024-09-16T16:15:48.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:48.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2024-09-16T16:15:49.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:49.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2024-09-16T16:15:49.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:49.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2024-09-16T16:15:49.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:49.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2024-09-16T16:15:50.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:50.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2024-09-16T16:15:50.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:50.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2024-09-16T16:15:51.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:51.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2024-09-16T16:15:51.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:51.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2024-09-16T16:15:51.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:51.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2024-09-16T16:15:52.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:52.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2024-09-16T16:15:52.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:52.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2024-09-16T16:15:52.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:52.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2024-09-16T16:15:53.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:53.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2024-09-16T16:15:54.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:54.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2024-09-16T16:15:54.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:54.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2024-09-16T16:15:54.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:54.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2024-09-16T16:15:55.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:55.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2024-09-16T16:15:55.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:55.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2024-09-16T16:15:55.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:55.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2024-09-16T16:15:56.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:56.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2024-09-16T16:15:56.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:56.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2024-09-16T16:15:57.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:57.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2024-09-16T16:15:57.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:57.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2024-09-16T16:15:57.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:57.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2024-09-16T16:15:58.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:58.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2024-09-16T16:15:58.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:58.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2024-09-16T16:15:59.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:59.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2024-09-16T16:15:59.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:59.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2024-09-16T16:15:59.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:15:59.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2024-09-16T16:16:00.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:00.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2024-09-16T16:16:00.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:00.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2024-09-16T16:16:01.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:01.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2024-09-16T16:16:01.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:01.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2024-09-16T16:16:01.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:01.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2024-09-16T16:16:02.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:02.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2024-09-16T16:16:02.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:02.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2024-09-16T16:16:02.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:02.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2024-09-16T16:16:03.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:03.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2024-09-16T16:16:03.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:03.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2024-09-16T16:16:04.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:04.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2024-09-16T16:16:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:04.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2024-09-16T16:16:04.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:04.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2024-09-16T16:16:05.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:05.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2024-09-16T16:16:05.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:05.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2024-09-16T16:16:06.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:06.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2024-09-16T16:16:06.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:06.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2024-09-16T16:16:06.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:06.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2024-09-16T16:16:07.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:07.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2024-09-16T16:16:07.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:07.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2024-09-16T16:16:08.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:08.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2024-09-16T16:16:08.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:08.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2024-09-16T16:16:08.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:08.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2024-09-16T16:16:09.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:09.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2024-09-16T16:16:09.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:09.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2024-09-16T16:16:10.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:10.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2024-09-16T16:16:10.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:10.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2024-09-16T16:16:10.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:10.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2024-09-16T16:16:11.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:11.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2024-09-16T16:16:11.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:11.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2024-09-16T16:16:11.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:11.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2024-09-16T16:16:12.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:12.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2024-09-16T16:16:12.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:12.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2024-09-16T16:16:13.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:13.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2024-09-16T16:16:13.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:13.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2024-09-16T16:16:13.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:13.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2024-09-16T16:16:14.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:14.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2024-09-16T16:16:14.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:14.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2024-09-16T16:16:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:15.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2024-09-16T16:16:15.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:15.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2024-09-16T16:16:15.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:15.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2024-09-16T16:16:16.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:16.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2024-09-16T16:16:16.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:16.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2024-09-16T16:16:17.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:17.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2024-09-16T16:16:17.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:17.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2024-09-16T16:16:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2024-09-16T16:16:18.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:18.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2024-09-16T16:16:18.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:18.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2024-09-16T16:16:19.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:19.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2024-09-16T16:16:19.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:19.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2024-09-16T16:16:19.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:19.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2024-09-16T16:16:20.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:20.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2024-09-16T16:16:20.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:20.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2024-09-16T16:16:21.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:21.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2024-09-16T16:16:21.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:21.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2024-09-16T16:16:21.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:21.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2024-09-16T16:16:22.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:22.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2024-09-16T16:16:22.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:22.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2024-09-16T16:16:23.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:23.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2024-09-16T16:16:23.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:23.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2024-09-16T16:16:23.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:23.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2024-09-16T16:16:24.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:24.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2024-09-16T16:16:24.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:24.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2024-09-16T16:16:25.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:25.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2024-09-16T16:16:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2024-09-16T16:16:26.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:26.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2024-09-16T16:16:26.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:26.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2024-09-16T16:16:26.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:26.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2024-09-16T16:16:27.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:27.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2024-09-16T16:16:27.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:27.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2024-09-16T16:16:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:28.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2024-09-16T16:16:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2024-09-16T16:16:28.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:28.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2024-09-16T16:16:29.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:29.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2024-09-16T16:16:29.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 2 + 1 2024-09-16T16:16:29.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=3 2024-09-16T16:16:29.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-09-16T16:16:29.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-09-16T16:16:29.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:29.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2024-09-16T16:16:30.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:30.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2024-09-16T16:16:30.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:30.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2024-09-16T16:16:30.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:30.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2024-09-16T16:16:31.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:31.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2024-09-16T16:16:31.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:31.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2024-09-16T16:16:32.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:32.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2024-09-16T16:16:32.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:32.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2024-09-16T16:16:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:32.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2024-09-16T16:16:33.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:33.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2024-09-16T16:16:33.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:33.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2024-09-16T16:16:34.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:34.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2024-09-16T16:16:34.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2024-09-16T16:16:34.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:34.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2024-09-16T16:16:35.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:35.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2024-09-16T16:16:35.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:35.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2024-09-16T16:16:36.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:36.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2024-09-16T16:16:36.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:36.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2024-09-16T16:16:36.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:36.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2024-09-16T16:16:37.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:37.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2024-09-16T16:16:37.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:37.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2024-09-16T16:16:38.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:38.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2024-09-16T16:16:38.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:38.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2024-09-16T16:16:38.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:38.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2024-09-16T16:16:39.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:39.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2024-09-16T16:16:39.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:39.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2024-09-16T16:16:40.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:40.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2024-09-16T16:16:40.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:40.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2024-09-16T16:16:40.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:40.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2024-09-16T16:16:41.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:41.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2024-09-16T16:16:41.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:41.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2024-09-16T16:16:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:42.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2024-09-16T16:16:42.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:42.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2024-09-16T16:16:42.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:42.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2024-09-16T16:16:43.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:43.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2024-09-16T16:16:43.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:43.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2024-09-16T16:16:44.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:44.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2024-09-16T16:16:44.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:44.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2024-09-16T16:16:44.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:44.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2024-09-16T16:16:45.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:45.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2024-09-16T16:16:45.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:45.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2024-09-16T16:16:45.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:45.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2024-09-16T16:16:46.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:46.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2024-09-16T16:16:46.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:46.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2024-09-16T16:16:47.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:47.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2024-09-16T16:16:47.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:47.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2024-09-16T16:16:47.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:47.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2024-09-16T16:16:48.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:48.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2024-09-16T16:16:48.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:48.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2024-09-16T16:16:48.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:48.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2024-09-16T16:16:49.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:49.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2024-09-16T16:16:49.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:49.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2024-09-16T16:16:50.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:50.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2024-09-16T16:16:50.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:50.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2024-09-16T16:16:50.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:50.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2024-09-16T16:16:51.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:51.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2024-09-16T16:16:51.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:51.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2024-09-16T16:16:52.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:52.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2024-09-16T16:16:52.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:52.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2024-09-16T16:16:52.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:52.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2024-09-16T16:16:53.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:53.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2024-09-16T16:16:53.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:53.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2024-09-16T16:16:53.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:53.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2024-09-16T16:16:54.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:54.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2024-09-16T16:16:54.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:54.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2024-09-16T16:16:55.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:55.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2024-09-16T16:16:55.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:55.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2024-09-16T16:16:56.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:56.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2024-09-16T16:16:56.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:56.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2024-09-16T16:16:56.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:56.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2024-09-16T16:16:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2024-09-16T16:16:57.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:57.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2024-09-16T16:16:58.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:58.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2024-09-16T16:16:58.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:58.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2024-09-16T16:16:58.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:58.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2024-09-16T16:16:59.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:59.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2024-09-16T16:16:59.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:16:59.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2024-09-16T16:17:00.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:00.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2024-09-16T16:17:00.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:00.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2024-09-16T16:17:00.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:00.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2024-09-16T16:17:01.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:01.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2024-09-16T16:17:01.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:01.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2024-09-16T16:17:02.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:02.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2024-09-16T16:17:02.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:02.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2024-09-16T16:17:02.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:02.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2024-09-16T16:17:03.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:03.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2024-09-16T16:17:03.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:03.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2024-09-16T16:17:04.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:04.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2024-09-16T16:17:04.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:04.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2024-09-16T16:17:04.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:04.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2024-09-16T16:17:05.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:05.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2024-09-16T16:17:05.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:05.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2024-09-16T16:17:06.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:06.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2024-09-16T16:17:06.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:06.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2024-09-16T16:17:06.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:06.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2024-09-16T16:17:07.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:07.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2024-09-16T16:17:07.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:07.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2024-09-16T16:17:08.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:08.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2024-09-16T16:17:08.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:08.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2024-09-16T16:17:09.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:09.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2024-09-16T16:17:09.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:09.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2024-09-16T16:17:09.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:09.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2024-09-16T16:17:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2024-09-16T16:17:10.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:10.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2024-09-16T16:17:11.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:11.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2024-09-16T16:17:11.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2024-09-16T16:17:11.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:11.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2024-09-16T16:17:12.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:12.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2024-09-16T16:17:12.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:12.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2024-09-16T16:17:13.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:13.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2024-09-16T16:17:13.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:13.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2024-09-16T16:17:13.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:13.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2024-09-16T16:17:14.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:14.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2024-09-16T16:17:14.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:14.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2024-09-16T16:17:15.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:15.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2024-09-16T16:17:15.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:15.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2024-09-16T16:17:15.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:15.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2024-09-16T16:17:16.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:16.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2024-09-16T16:17:16.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:16.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2024-09-16T16:17:16.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:16.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2024-09-16T16:17:17.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:17.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2024-09-16T16:17:17.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:17.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2024-09-16T16:17:18.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:18.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2024-09-16T16:17:18.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:18.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2024-09-16T16:17:18.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:18.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2024-09-16T16:17:19.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:19.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2024-09-16T16:17:19.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:19.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2024-09-16T16:17:20.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:20.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2024-09-16T16:17:20.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:20.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2024-09-16T16:17:20.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:20.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2024-09-16T16:17:21.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:21.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2024-09-16T16:17:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:21.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2024-09-16T16:17:22.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:22.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2024-09-16T16:17:22.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:22.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2024-09-16T16:17:22.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:22.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2024-09-16T16:17:23.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:23.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2024-09-16T16:17:23.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:23.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2024-09-16T16:17:24.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:24.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2024-09-16T16:17:24.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:24.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2024-09-16T16:17:24.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:24.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2024-09-16T16:17:25.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:25.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2024-09-16T16:17:25.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:25.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2024-09-16T16:17:26.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:26.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2024-09-16T16:17:26.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:26.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2024-09-16T16:17:27.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:27.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2024-09-16T16:17:27.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:27.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2024-09-16T16:17:27.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:27.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2024-09-16T16:17:28.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:28.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2024-09-16T16:17:28.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:28.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2024-09-16T16:17:29.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:29.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2024-09-16T16:17:29.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:29.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2024-09-16T16:17:29.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:29.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2024-09-16T16:17:30.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:30.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2024-09-16T16:17:30.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:30.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2024-09-16T16:17:31.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:31.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2024-09-16T16:17:31.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:31.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2024-09-16T16:17:31.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:31.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2024-09-16T16:17:32.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:32.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2024-09-16T16:17:32.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:32.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2024-09-16T16:17:33.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:33.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2024-09-16T16:17:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2024-09-16T16:17:33.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:33.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2024-09-16T16:17:34.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:34.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2024-09-16T16:17:34.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:34.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2024-09-16T16:17:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:35.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2024-09-16T16:17:35.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:35.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2024-09-16T16:17:35.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:35.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2024-09-16T16:17:36.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:36.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2024-09-16T16:17:36.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:36.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2024-09-16T16:17:36.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:36.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2024-09-16T16:17:37.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:37.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2024-09-16T16:17:37.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:37.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2024-09-16T16:17:38.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:38.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2024-09-16T16:17:38.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:38.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2024-09-16T16:17:38.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:38.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2024-09-16T16:17:39.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2024-09-16T16:17:39.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:39.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2024-09-16T16:17:40.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:40.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2024-09-16T16:17:40.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:40.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2024-09-16T16:17:40.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:40.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2024-09-16T16:17:41.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:41.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2024-09-16T16:17:41.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:41.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2024-09-16T16:17:42.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:42.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2024-09-16T16:17:42.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:42.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2024-09-16T16:17:42.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:42.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2024-09-16T16:17:43.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:43.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2024-09-16T16:17:43.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:43.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2024-09-16T16:17:44.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:44.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2024-09-16T16:17:44.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:44.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2024-09-16T16:17:44.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:44.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2024-09-16T16:17:45.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:45.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2024-09-16T16:17:45.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:45.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2024-09-16T16:17:45.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:45.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2024-09-16T16:17:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2024-09-16T16:17:46.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:46.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2024-09-16T16:17:47.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:47.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2024-09-16T16:17:47.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:47.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2024-09-16T16:17:47.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:47.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2024-09-16T16:17:48.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:48.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2024-09-16T16:17:48.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-16T16:17:48.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2024-09-16T16:17:48.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 3 + 1 2024-09-16T16:17:48.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=4 2024-09-16T16:17:48.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: get_not_primary test1 obj1-p1 2024-09-16T16:17:48.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-09-16T16:17:48.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-09-16T16:17:48.967 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-09-16T16:17:48.967 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-09-16T16:17:48.967 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-09-16T16:17:48.968 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-09-16T16:17:48.968 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:17:49.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-16T16:17:49.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-09-16T16:17:49.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-16T16:17:49.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: local otherosd=0 2024-09-16T16:17:49.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph pg dump pgs 2024-09-16T16:17:50.377 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T16:17:50.378 INFO:tasks.workunit.client.0.smithi039.stdout:7.0 198 0 0 0 0 2076180480 0 0 594 0 594 active+clean 2024-09-16T16:12:47.539025+0000 96'594 96:646 [1] 1 [1] 1 0'0 2024-09-16T16:12:15.456943+0000 0'0 2024-09-16T16:12:15.456943+0000 0 0 periodic scrub scheduled @ 2024-09-17T19:34:55.687390+0000 0 0 2024-09-16T16:17:50.378 INFO:tasks.workunit.client.0.smithi039.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-16T16:12:46.347765+0000 96'600 96:668 [1] 1 [1] 1 0'0 2024-09-16T16:12:03.866975+0000 0'0 2024-09-16T16:12:03.866975+0000 0 0 periodic scrub scheduled @ 2024-09-18T01:48:38.891658+0000 0 0 2024-09-16T16:17:50.378 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-16T16:12:01.512213+0000 96'600 96:662 [1,0] 1 [1,0] 1 0'0 2024-09-16T16:11:58.123419+0000 0'0 2024-09-16T16:11:58.123419+0000 0 0 periodic scrub scheduled @ 2024-09-17T16:58:06.592371+0000 0 0 2024-09-16T16:17:50.378 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:17:50.378 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T16:17:50.378 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:17:50.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ERRORS=0 2024-09-16T16:17:50.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:151: TEST_recovery_priority: ceph osd set norecover 2024-09-16T16:17:51.271 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is set 2024-09-16T16:17:51.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:152: TEST_recovery_priority: ceph osd set noout 2024-09-16T16:17:51.973 INFO:tasks.workunit.client.0.smithi039.stderr:noout is set 2024-09-16T16:17:51.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:156: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-09-16T16:17:53.211 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 7 size to 2 2024-09-16T16:17:53.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:157: TEST_recovery_priority: sleep 2 2024-09-16T16:17:55.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: get_asok_path osd.1 2024-09-16T16:17:55.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T16:17:55.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T16:17:55.239 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:17:55.240 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:17:55.240 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:17:55.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.1.asok 2024-09-16T16:17:55.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: CEPH_ARGS= 2024-09-16T16:17:55.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:17:55.359 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:17:55.359 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:17:55.359 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:17:55.359 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:17:55.360 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:17:55.360 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:17:55.360 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:17:55.360 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:17:55.360 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:17:55.360 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:17:55.360 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:17:55.360 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:17:55.360 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:17:55.360 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:17:55.361 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:17:55.361 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:17:55.361 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:17:55.361 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:17:55.361 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:17:55.361 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:17:55.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: seq 1 10 2024-09-16T16:17:55.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-09-16T16:17:55.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-09-16T16:17:55.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-09-16T16:17:55.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 1 = 10 ']' 2024-09-16T16:17:55.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-09-16T16:17:57.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-09-16T16:17:57.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-09-16T16:17:57.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-09-16T16:17:58.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 2 = 10 ']' 2024-09-16T16:17:58.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-09-16T16:18:00.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-09-16T16:18:00.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-09-16T16:18:00.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-09-16T16:18:00.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:164: TEST_recovery_priority: break 2024-09-16T16:18:00.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: flush_pg_stats 2024-09-16T16:18:00.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:18:00.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:18:01.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:18:01.131 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:18:01.131 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:18:01.131 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:18:01.131 INFO:tasks.workunit.client.0.smithi039.stderr:4' 2024-09-16T16:18:01.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:18:01.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:01.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:18:01.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836570 2024-09-16T16:18:01.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836570 2024-09-16T16:18:01.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836570' 2024-09-16T16:18:01.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:01.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:18:01.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673047 2024-09-16T16:18:01.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673047 2024-09-16T16:18:01.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836570 1-42949673047' 2024-09-16T16:18:01.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:01.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:18:01.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509525 2024-09-16T16:18:01.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509525 2024-09-16T16:18:01.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836570 1-42949673047 2-64424509525' 2024-09-16T16:18:01.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:01.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:18:01.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899346002 2024-09-16T16:18:01.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899346002 2024-09-16T16:18:01.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836570 1-42949673047 2-64424509525 3-85899346002' 2024-09-16T16:18:01.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:01.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:18:01.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182479 2024-09-16T16:18:01.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182479 2024-09-16T16:18:01.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836570 1-42949673047 2-64424509525 3-85899346002 4-107374182479' 2024-09-16T16:18:01.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:01.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836570 2024-09-16T16:18:01.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:01.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:18:01.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836570 2024-09-16T16:18:01.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:01.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836570 2024-09-16T16:18:01.942 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836570 2024-09-16T16:18:01.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-09-16T16:18:01.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:18:02.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836568 -lt 21474836570 2024-09-16T16:18:02.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:18:03.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:18:03.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:18:03.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836570 -lt 21474836570 2024-09-16T16:18:03.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:03.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673047 2024-09-16T16:18:03.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:03.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:18:03.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673047 2024-09-16T16:18:03.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:03.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673047 2024-09-16T16:18:03.798 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673047 2024-09-16T16:18:03.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-09-16T16:18:03.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:18:04.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673047 -lt 42949673047 2024-09-16T16:18:04.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:04.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509525 2024-09-16T16:18:04.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:04.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:18:04.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509525 2024-09-16T16:18:04.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:04.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509525 2024-09-16T16:18:04.233 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509525 2024-09-16T16:18:04.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-09-16T16:18:04.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:18:04.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509525 -lt 64424509525 2024-09-16T16:18:04.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:04.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:04.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899346002 2024-09-16T16:18:04.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:18:04.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899346002 2024-09-16T16:18:04.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:04.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899346002 2024-09-16T16:18:04.665 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899346002 2024-09-16T16:18:04.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899346002' 2024-09-16T16:18:04.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:18:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346003 -lt 85899346002 2024-09-16T16:18:05.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:05.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182479 2024-09-16T16:18:05.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:05.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:18:05.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182479 2024-09-16T16:18:05.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:05.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182479 2024-09-16T16:18:05.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-09-16T16:18:05.129 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182479 2024-09-16T16:18:05.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:18:05.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182480 -lt 107374182479 2024-09-16T16:18:05.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: get_asok_path osd.1 2024-09-16T16:18:05.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T16:18:05.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T16:18:05.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:18:05.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:18:05.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:18:05.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.1.asok 2024-09-16T16:18:05.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: CEPH_ARGS= 2024-09-16T16:18:05.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:05.698 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:18:05.698 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:18:05.698 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:05.698 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:05.698 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:05.698 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:18:05.698 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:05.698 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:18:05.698 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:18:05.699 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:05.699 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:05.699 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:05.699 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:05.699 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:18:05.699 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:05.699 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:05.699 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:05.699 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:18:05.699 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:05.699 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:18:05.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph osd out osd.0 2024-09-16T16:18:06.273 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T16:18:06.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: sleep 2 2024-09-16T16:18:08.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:177: TEST_recovery_priority: flush_pg_stats 2024-09-16T16:18:08.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:18:08.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:18:08.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:18:08.730 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:18:08.730 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:18:08.730 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:18:08.730 INFO:tasks.workunit.client.0.smithi039.stderr:4' 2024-09-16T16:18:08.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:18:08.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:08.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:18:08.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836573 2024-09-16T16:18:08.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836573 2024-09-16T16:18:08.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836573' 2024-09-16T16:18:08.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:08.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:18:09.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673051 2024-09-16T16:18:09.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673051 2024-09-16T16:18:09.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836573 1-42949673051' 2024-09-16T16:18:09.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:09.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:18:09.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509528 2024-09-16T16:18:09.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509528 2024-09-16T16:18:09.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509528' 2024-09-16T16:18:09.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:09.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:18:09.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899346006 2024-09-16T16:18:09.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899346006 2024-09-16T16:18:09.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509528 3-85899346006' 2024-09-16T16:18:09.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:09.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:18:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182483 2024-09-16T16:18:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182483 2024-09-16T16:18:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509528 3-85899346006 4-107374182483' 2024-09-16T16:18:09.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:09.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836573 2024-09-16T16:18:09.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:09.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:18:09.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836573 2024-09-16T16:18:09.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:09.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836573 2024-09-16T16:18:09.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-09-16T16:18:09.538 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836573 2024-09-16T16:18:09.539 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:18:09.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836573 -lt 21474836573 2024-09-16T16:18:09.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:09.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673051 2024-09-16T16:18:09.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:09.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:18:09.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673051 2024-09-16T16:18:09.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:09.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673051 2024-09-16T16:18:09.951 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673051 2024-09-16T16:18:09.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-09-16T16:18:09.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:18:10.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673049 -lt 42949673051 2024-09-16T16:18:10.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:18:11.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:18:11.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:18:11.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673051 -lt 42949673051 2024-09-16T16:18:11.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:11.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509528 2024-09-16T16:18:11.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:11.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:18:11.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509528 2024-09-16T16:18:11.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:11.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509528 2024-09-16T16:18:11.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-09-16T16:18:11.801 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509528 2024-09-16T16:18:11.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:18:12.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509528 -lt 64424509528 2024-09-16T16:18:12.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:12.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899346006 2024-09-16T16:18:12.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:12.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:18:12.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899346006 2024-09-16T16:18:12.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:12.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899346006 2024-09-16T16:18:12.223 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899346006 2024-09-16T16:18:12.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899346006' 2024-09-16T16:18:12.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:18:12.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346006 -lt 85899346006 2024-09-16T16:18:12.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:12.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182483 2024-09-16T16:18:12.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:12.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:18:12.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182483 2024-09-16T16:18:12.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:12.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182483 2024-09-16T16:18:12.659 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182483 2024-09-16T16:18:12.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182483' 2024-09-16T16:18:12.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:18:13.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182484 -lt 107374182483 2024-09-16T16:18:13.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: get_asok_path osd.1 2024-09-16T16:18:13.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T16:18:13.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T16:18:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:18:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:18:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:18:13.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.1.asok 2024-09-16T16:18:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: CEPH_ARGS= 2024-09-16T16:18:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:13.179 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:18:13.179 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:18:13.179 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:13.180 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:13.182 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:18:13.182 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:13.182 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:18:13.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: ceph pg dump pgs 2024-09-16T16:18:13.596 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T16:18:13.596 INFO:tasks.workunit.client.0.smithi039.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-09-16T16:18:00.514914+0000 96'600 107:672 [1,4] 1 [1] 1 0'0 2024-09-16T16:12:15.456943+0000 0'0 2024-09-16T16:12:15.456943+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:02:41.576424+0000 0 0 2024-09-16T16:18:13.596 INFO:tasks.workunit.client.0.smithi039.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-16T16:12:46.347765+0000 96'600 108:680 [1] 1 [1] 1 0'0 2024-09-16T16:12:03.866975+0000 0'0 2024-09-16T16:12:03.866975+0000 0 0 periodic scrub scheduled @ 2024-09-18T01:48:38.891658+0000 0 0 2024-09-16T16:18:13.597 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-09-16T16:18:07.949352+0000 96'600 108:681 [1,2] 1 [1] 1 0'0 2024-09-16T16:11:58.123419+0000 0'0 2024-09-16T16:11:58.123419+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:13:07.374349+0000 0 0 2024-09-16T16:18:13.597 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:18:13.597 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T16:18:13.597 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:18:13.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-09-16T16:18:14.331 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 3 size to 2 2024-09-16T16:18:14.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: sleep 2 2024-09-16T16:18:16.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: flush_pg_stats 2024-09-16T16:18:16.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:18:16.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:18:16.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:18:16.791 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:18:16.791 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:18:16.791 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:18:16.791 INFO:tasks.workunit.client.0.smithi039.stderr:4' 2024-09-16T16:18:16.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:18:16.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:16.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:18:16.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836577 2024-09-16T16:18:16.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836577 2024-09-16T16:18:16.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836577' 2024-09-16T16:18:16.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:16.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:18:17.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673054 2024-09-16T16:18:17.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673054 2024-09-16T16:18:17.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836577 1-42949673054' 2024-09-16T16:18:17.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:17.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:18:17.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509532 2024-09-16T16:18:17.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509532 2024-09-16T16:18:17.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509532' 2024-09-16T16:18:17.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:17.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:18:17.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899346009 2024-09-16T16:18:17.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899346009 2024-09-16T16:18:17.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509532 3-85899346009' 2024-09-16T16:18:17.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:17.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:18:17.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182487 2024-09-16T16:18:17.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182487 2024-09-16T16:18:17.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509532 3-85899346009 4-107374182487' 2024-09-16T16:18:17.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:17.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836577 2024-09-16T16:18:17.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:17.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:18:17.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836577 2024-09-16T16:18:17.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:17.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836577 2024-09-16T16:18:17.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-09-16T16:18:17.598 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836577 2024-09-16T16:18:17.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:18:18.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836575 -lt 21474836577 2024-09-16T16:18:18.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:18:19.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:18:19.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:18:19.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836577 -lt 21474836577 2024-09-16T16:18:19.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:19.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673054 2024-09-16T16:18:19.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:19.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:18:19.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673054 2024-09-16T16:18:19.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:19.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673054 2024-09-16T16:18:19.442 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673054 2024-09-16T16:18:19.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-09-16T16:18:19.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:18:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673055 -lt 42949673054 2024-09-16T16:18:19.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:19.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509532 2024-09-16T16:18:19.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:19.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:18:19.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509532 2024-09-16T16:18:19.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:19.879 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509532 2024-09-16T16:18:19.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509532 2024-09-16T16:18:19.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-09-16T16:18:19.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:18:20.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509532 -lt 64424509532 2024-09-16T16:18:20.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:20.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:20.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899346009 2024-09-16T16:18:20.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:18:20.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899346009 2024-09-16T16:18:20.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:20.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899346009 2024-09-16T16:18:20.354 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899346009 2024-09-16T16:18:20.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899346009' 2024-09-16T16:18:20.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:18:20.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346009 -lt 85899346009 2024-09-16T16:18:20.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:20.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182487 2024-09-16T16:18:20.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:20.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:18:20.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182487 2024-09-16T16:18:20.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:20.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182487 2024-09-16T16:18:20.784 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182487 2024-09-16T16:18:20.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2024-09-16T16:18:20.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:18:21.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182487 -lt 107374182487 2024-09-16T16:18:21.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: get_asok_path osd.1 2024-09-16T16:18:21.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T16:18:21.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T16:18:21.220 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:18:21.220 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:18:21.220 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:18:21.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.1.asok 2024-09-16T16:18:21.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: CEPH_ARGS= 2024-09-16T16:18:21.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:21.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:21.342 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:18:21.342 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:18:21.342 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:21.342 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:21.342 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:18:21.342 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:21.343 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:18:21.343 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:18:21.343 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:21.343 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:18:21.343 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:21.343 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:21.343 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:21.343 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:21.343 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:18:21.343 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:21.344 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:21.344 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:21.344 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:21.344 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:21.344 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:18:21.344 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:18:21.344 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:21.344 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:18:21.344 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:18:21.344 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:21.345 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:21.345 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:21.345 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:21.345 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:18:21.345 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:21.345 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:21.345 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:21.345 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:18:21.345 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:21.346 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:18:21.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: ceph pg dump pgs 2024-09-16T16:18:21.733 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T16:18:21.733 INFO:tasks.workunit.client.0.smithi039.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-09-16T16:18:00.514914+0000 96'600 113:677 [1,4] 1 [1] 1 0'0 2024-09-16T16:12:15.456943+0000 0'0 2024-09-16T16:12:15.456943+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:02:41.576424+0000 0 0 2024-09-16T16:18:21.733 INFO:tasks.workunit.client.0.smithi039.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-09-16T16:18:15.730426+0000 96'600 113:691 [1,2] 1 [1] 1 0'0 2024-09-16T16:12:03.866975+0000 0'0 2024-09-16T16:12:03.866975+0000 0 0 periodic scrub scheduled @ 2024-09-17T19:51:22.624878+0000 0 0 2024-09-16T16:18:21.733 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-09-16T16:18:07.949352+0000 96'600 112:685 [1,2] 1 [1] 1 0'0 2024-09-16T16:11:58.123419+0000 0'0 2024-09-16T16:11:58.123419+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:13:07.374349+0000 0 0 2024-09-16T16:18:21.733 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:18:21.734 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T16:18:21.734 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:18:21.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:21.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2024-09-16T16:18:21.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: PRIO=190 2024-09-16T16:18:21.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-09-16T16:18:21.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:21.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-09-16T16:18:21.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-09-16T16:18:21.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: ITEM=7.0 2024-09-16T16:18:21.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-09-16T16:18:21.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:21.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-16T16:18:21.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: PRIO=255 2024-09-16T16:18:21.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:203: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-16T16:18:21.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: seq 1 10 2024-09-16T16:18:21.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-09-16T16:18:21.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: ceph pg force-recovery 3.0 2024-09-16T16:18:21.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-09-16T16:18:22.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:214: TEST_recovery_priority: break 2024-09-16T16:18:22.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: sleep 2 2024-09-16T16:18:24.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: get_asok_path osd.1 2024-09-16T16:18:24.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T16:18:24.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T16:18:24.289 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:18:24.289 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:18:24.290 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:18:24.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.1.asok 2024-09-16T16:18:24.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: CEPH_ARGS= 2024-09-16T16:18:24.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:24.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:224: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:24.421 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:18:24.422 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:18:24.422 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:24.422 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:24.422 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:18:24.422 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:24.422 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:18:24.422 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:18:24.423 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:24.423 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:18:24.423 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:24.423 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:24.423 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:24.423 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:24.423 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:24.423 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:24.423 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 255, 2024-09-16T16:18:24.423 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:18:24.424 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:24.424 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:18:24.424 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:18:24.424 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:24.424 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:24.424 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:24.424 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:24.424 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:18:24.424 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:18:24.425 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:24.425 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:18:24.425 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:18:24.425 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:24.425 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:24.425 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:24.425 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:24.425 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:18:24.425 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:24.426 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:24.426 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:24.426 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:18:24.426 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:24.426 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:18:24.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:24.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-09-16T16:18:24.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: PRIO=255 2024-09-16T16:18:24.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-16T16:18:24.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:231: TEST_recovery_priority: flush_pg_stats 2024-09-16T16:18:24.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:18:24.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:18:24.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:18:24.888 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:18:24.888 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:18:24.888 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:18:24.888 INFO:tasks.workunit.client.0.smithi039.stderr:4' 2024-09-16T16:18:24.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:18:24.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:24.889 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:18:25.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836580 2024-09-16T16:18:25.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836580 2024-09-16T16:18:25.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580' 2024-09-16T16:18:25.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:25.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:18:25.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673058 2024-09-16T16:18:25.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673058 2024-09-16T16:18:25.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673058' 2024-09-16T16:18:25.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:25.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:18:25.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509535 2024-09-16T16:18:25.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509535 2024-09-16T16:18:25.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509535' 2024-09-16T16:18:25.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:25.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:18:25.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899346013 2024-09-16T16:18:25.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899346013 2024-09-16T16:18:25.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509535 3-85899346013' 2024-09-16T16:18:25.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:25.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:18:25.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182490 2024-09-16T16:18:25.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182490 2024-09-16T16:18:25.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509535 3-85899346013 4-107374182490' 2024-09-16T16:18:25.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:25.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836580 2024-09-16T16:18:25.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:25.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:18:25.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836580 2024-09-16T16:18:25.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:25.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836580 2024-09-16T16:18:25.657 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836580 2024-09-16T16:18:25.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-09-16T16:18:25.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:18:26.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836578 -lt 21474836580 2024-09-16T16:18:26.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:18:27.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:18:27.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:18:27.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836581 -lt 21474836580 2024-09-16T16:18:27.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:27.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:27.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673058 2024-09-16T16:18:27.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:18:27.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673058 2024-09-16T16:18:27.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:27.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673058 2024-09-16T16:18:27.512 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673058 2024-09-16T16:18:27.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-09-16T16:18:27.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:18:28.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673058 -lt 42949673058 2024-09-16T16:18:28.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:28.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509535 2024-09-16T16:18:28.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:28.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:18:28.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509535 2024-09-16T16:18:28.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:28.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509535 2024-09-16T16:18:28.125 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509535 2024-09-16T16:18:28.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-09-16T16:18:28.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:18:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509536 -lt 64424509535 2024-09-16T16:18:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:28.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899346013 2024-09-16T16:18:28.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:18:28.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899346013 2024-09-16T16:18:28.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:28.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899346013 2024-09-16T16:18:28.558 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899346013 2024-09-16T16:18:28.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899346013' 2024-09-16T16:18:28.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:18:29.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346013 -lt 85899346013 2024-09-16T16:18:29.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:29.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182490 2024-09-16T16:18:29.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:29.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:18:29.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182490 2024-09-16T16:18:29.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:29.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182490 2024-09-16T16:18:29.071 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182490 2024-09-16T16:18:29.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182490' 2024-09-16T16:18:29.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:18:29.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182491 -lt 107374182490 2024-09-16T16:18:29.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2024-09-16T16:18:29.970 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2024-09-16T16:18:29.970 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T16:18:29.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: sleep 2 2024-09-16T16:18:31.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: get_asok_path osd.1 2024-09-16T16:18:31.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T16:18:31.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T16:18:31.992 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:18:31.992 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:18:31.992 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:18:31.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.1.asok 2024-09-16T16:18:31.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: CEPH_ARGS= 2024-09-16T16:18:31.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:32.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:239: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:32.126 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:18:32.126 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:18:32.126 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:32.126 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:32.126 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:18:32.126 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:32.126 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:18:32.126 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:18:32.126 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:32.126 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:32.128 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:18:32.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:32.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2024-09-16T16:18:32.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: PRIO=190 2024-09-16T16:18:32.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-09-16T16:18:32.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:32.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-09-16T16:18:32.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: eval 'ITEM="3.0"' 2024-09-16T16:18:32.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: ITEM=3.0 2024-09-16T16:18:32.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: '[' 3.0 '!=' 3.0 ']' 2024-09-16T16:18:32.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:32.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-16T16:18:32.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: PRIO=255 2024-09-16T16:18:32.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:254: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-16T16:18:32.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:261: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2024-09-16T16:18:32.623 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2024-09-16T16:18:32.623 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T16:18:32.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:262: TEST_recovery_priority: sleep 5 2024-09-16T16:18:37.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: get_asok_path osd.1 2024-09-16T16:18:37.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T16:18:37.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T16:18:37.642 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:18:37.643 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:18:37.643 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:18:37.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.1.asok 2024-09-16T16:18:37.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: CEPH_ARGS= 2024-09-16T16:18:37.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:37.763 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:18:37.763 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:18:37.763 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:37.763 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:37.764 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:18:37.765 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:37.766 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:37.766 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:37.766 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:18:37.766 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:37.766 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:18:37.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:266: TEST_recovery_priority: flush_pg_stats 2024-09-16T16:18:37.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:18:37.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:18:38.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:18:38.237 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:18:38.237 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:18:38.237 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:18:38.238 INFO:tasks.workunit.client.0.smithi039.stderr:4' 2024-09-16T16:18:38.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:18:38.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:38.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:18:38.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836585 2024-09-16T16:18:38.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836585 2024-09-16T16:18:38.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836585' 2024-09-16T16:18:38.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:38.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:18:38.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673063 2024-09-16T16:18:38.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673063 2024-09-16T16:18:38.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836585 1-42949673063' 2024-09-16T16:18:38.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:38.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:18:38.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509540 2024-09-16T16:18:38.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509540 2024-09-16T16:18:38.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836585 1-42949673063 2-64424509540' 2024-09-16T16:18:38.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:38.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:18:38.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899346017 2024-09-16T16:18:38.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899346017 2024-09-16T16:18:38.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836585 1-42949673063 2-64424509540 3-85899346017' 2024-09-16T16:18:38.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:38.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:18:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182495 2024-09-16T16:18:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182495 2024-09-16T16:18:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836585 1-42949673063 2-64424509540 3-85899346017 4-107374182495' 2024-09-16T16:18:39.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:39.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836585 2024-09-16T16:18:39.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:39.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:18:39.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836585 2024-09-16T16:18:39.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:39.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836585 2024-09-16T16:18:39.057 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836585 2024-09-16T16:18:39.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-09-16T16:18:39.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:18:39.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836585 -lt 21474836585 2024-09-16T16:18:39.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:39.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673063 2024-09-16T16:18:39.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:39.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:18:39.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673063 2024-09-16T16:18:39.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:39.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673063 2024-09-16T16:18:39.519 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673063 2024-09-16T16:18:39.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-09-16T16:18:39.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:18:39.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673063 -lt 42949673063 2024-09-16T16:18:39.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:39.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509540 2024-09-16T16:18:39.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:39.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:18:39.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509540 2024-09-16T16:18:39.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:39.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509540 2024-09-16T16:18:39.966 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509540 2024-09-16T16:18:39.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509540' 2024-09-16T16:18:39.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:18:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509540 -lt 64424509540 2024-09-16T16:18:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:40.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899346017 2024-09-16T16:18:40.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:40.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:18:40.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899346017 2024-09-16T16:18:40.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:40.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899346017 2024-09-16T16:18:40.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899346017' 2024-09-16T16:18:40.389 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899346017 2024-09-16T16:18:40.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:18:40.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346017 -lt 85899346017 2024-09-16T16:18:40.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:40.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182495 2024-09-16T16:18:40.837 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:40.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:18:40.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182495 2024-09-16T16:18:40.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:40.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182495 2024-09-16T16:18:40.840 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182495 2024-09-16T16:18:40.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182495' 2024-09-16T16:18:40.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:18:41.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182495 -lt 107374182495 2024-09-16T16:18:41.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-09-16T16:18:41.758 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2024-09-16T16:18:41.759 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-16T16:18:41.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: sleep 2 2024-09-16T16:18:43.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: get_asok_path osd.1 2024-09-16T16:18:43.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T16:18:43.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T16:18:43.779 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:18:43.780 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:18:43.780 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:18:43.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.1.asok 2024-09-16T16:18:43.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: CEPH_ARGS= 2024-09-16T16:18:43.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:43.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:271: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:43.914 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:18:43.915 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:18:43.915 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:43.915 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:43.915 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:18:43.915 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:43.915 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:18:43.915 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:18:43.915 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:43.915 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:18:43.915 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:43.915 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:43.916 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:43.916 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:43.916 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:18:43.916 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:43.916 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:43.916 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:43.916 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:43.916 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:43.916 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:18:43.916 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:18:43.916 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:43.917 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:18:43.917 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:18:43.917 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:43.917 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:43.917 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:43.917 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:43.917 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:18:43.917 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:43.917 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:43.917 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:43.917 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:18:43.918 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:43.918 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:18:43.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:43.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-09-16T16:18:43.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: PRIO=190 2024-09-16T16:18:43.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-09-16T16:18:43.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:43.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-09-16T16:18:43.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-09-16T16:18:43.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: ITEM=7.0 2024-09-16T16:18:43.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-09-16T16:18:43.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-16T16:18:43.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-16T16:18:44.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: PRIO=255 2024-09-16T16:18:44.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:286: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-16T16:18:44.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: ceph osd unset noout 2024-09-16T16:18:44.756 INFO:tasks.workunit.client.0.smithi039.stderr:noout is unset 2024-09-16T16:18:44.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:294: TEST_recovery_priority: ceph osd unset norecover 2024-09-16T16:18:45.576 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is unset 2024-09-16T16:18:45.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: get_asok_path osd.1 2024-09-16T16:18:45.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T16:18:45.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T16:18:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:18:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:18:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:18:45.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.1.asok 2024-09-16T16:18:45.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations' 2024-09-16T16:18:45.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations' 2024-09-16T16:18:45.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:18:45.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:18:45.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:18:45.601 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:18:45.602 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:18:45.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:18:45.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:18:45.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:18:45.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:18:45.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:18:45.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:18:45.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:18:45.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:18:45.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:18:46.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:18:46.414 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:18:46.414 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:18:46.414 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:18:46.414 INFO:tasks.workunit.client.0.smithi039.stderr:4' 2024-09-16T16:18:46.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:18:46.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:46.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:18:46.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836589 2024-09-16T16:18:46.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836589 2024-09-16T16:18:46.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836589' 2024-09-16T16:18:46.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:46.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:18:46.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673066 2024-09-16T16:18:46.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673066 2024-09-16T16:18:46.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836589 1-42949673066' 2024-09-16T16:18:46.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:46.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:18:46.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509544 2024-09-16T16:18:46.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509544 2024-09-16T16:18:46.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836589 1-42949673066 2-64424509544' 2024-09-16T16:18:46.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:46.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:18:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899346021 2024-09-16T16:18:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899346021 2024-09-16T16:18:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836589 1-42949673066 2-64424509544 3-85899346021' 2024-09-16T16:18:47.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:18:47.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:18:47.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182498 2024-09-16T16:18:47.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182498 2024-09-16T16:18:47.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836589 1-42949673066 2-64424509544 3-85899346021 4-107374182498' 2024-09-16T16:18:47.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:47.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836589 2024-09-16T16:18:47.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:47.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:18:47.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836589 2024-09-16T16:18:47.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:47.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836589 2024-09-16T16:18:47.210 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836589 2024-09-16T16:18:47.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-09-16T16:18:47.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:18:47.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836589 -lt 21474836589 2024-09-16T16:18:47.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:47.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673066 2024-09-16T16:18:47.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:47.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:18:47.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673066 2024-09-16T16:18:47.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:47.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673066 2024-09-16T16:18:47.689 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673066 2024-09-16T16:18:47.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673066' 2024-09-16T16:18:47.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:18:48.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673066 -lt 42949673066 2024-09-16T16:18:48.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:48.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509544 2024-09-16T16:18:48.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:48.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:18:48.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509544 2024-09-16T16:18:48.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:48.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509544 2024-09-16T16:18:48.136 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509544 2024-09-16T16:18:48.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509544' 2024-09-16T16:18:48.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:18:48.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509544 -lt 64424509544 2024-09-16T16:18:48.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:48.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:48.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899346021 2024-09-16T16:18:48.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:18:48.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899346021 2024-09-16T16:18:48.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:48.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899346021 2024-09-16T16:18:48.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899346021' 2024-09-16T16:18:48.707 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899346021 2024-09-16T16:18:48.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:18:49.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346019 -lt 85899346021 2024-09-16T16:18:49.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:18:50.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:18:50.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:18:50.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346021 -lt 85899346021 2024-09-16T16:18:50.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:18:50.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182498 2024-09-16T16:18:50.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:18:50.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:18:50.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182498 2024-09-16T16:18:50.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:18:50.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182498 2024-09-16T16:18:50.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182498' 2024-09-16T16:18:50.629 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182498 2024-09-16T16:18:50.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:18:51.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182499 -lt 107374182498 2024-09-16T16:18:51.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:18:51.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:18:51.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:18:51.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-09-16T16:18:51.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:18:51.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:18:51.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:18:51.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:18:51.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:18:51.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:18:51.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:18:52.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:18:52.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:18:52.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:18:52.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:18:52.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:18:52.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T16:18:52.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T16:18:52.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T16:18:52.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:52.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:18:52.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:52.839 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:18:52.840 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:18:52.840 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:52.840 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:52.840 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:18:52.840 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:52.840 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:18:52.840 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:18:52.840 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:52.840 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:18:52.840 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:52.841 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:52.841 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:52.841 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:52.841 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:18:52.841 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:52.841 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:52.841 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:52.841 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:52.841 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:52.841 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:18:52.841 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:18:52.842 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:52.842 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:18:52.842 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:18:52.842 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:52.842 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:52.842 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:52.842 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:52.842 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:18:52.842 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:52.842 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:52.842 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:52.843 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:18:52.843 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:52.843 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:18:52.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:18:52.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:18:52.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:18:52.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:18:52.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:18:52.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:18:52.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:18:52.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:18:52.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:18:53.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:18:53.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:18:53.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:18:53.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:18:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:18:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:18:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:18:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:18:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:18:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:18:54.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:18:54.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:18:54.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:18:54.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5241620 2024-09-16T16:18:54.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5241620 '!=' null 2024-09-16T16:18:54.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:18:54.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:54.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:18:54.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:54.653 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:18:54.653 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:18:54.653 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:54.653 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:54.653 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:18:54.653 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:54.653 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:18:54.653 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:18:54.654 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:54.655 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:18:54.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:18:54.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:18:54.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:18:54.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:18:54.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:18:54.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:18:54.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:18:54.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:18:54.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:18:55.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:18:55.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:18:55.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:18:55.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:18:55.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:18:55.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:18:55.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:18:55.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:18:55.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:18:55.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:18:55.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:18:55.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:18:55.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:18:56.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15377431 2024-09-16T16:18:56.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15377431 '!=' null 2024-09-16T16:18:56.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:18:56.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:56.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:18:56.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:56.581 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:18:56.581 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:18:56.581 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:56.581 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:56.581 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:18:56.581 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:56.581 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:18:56.581 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:18:56.581 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:56.581 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:18:56.581 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:18:56.582 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:18:56.583 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:56.583 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:56.583 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:56.583 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:56.583 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:18:56.583 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:56.583 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:56.583 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:56.583 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:18:56.583 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:56.583 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:18:56.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:18:56.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:18:56.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:18:56.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:18:56.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:18:56.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:18:56.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:18:56.701 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:18:56.701 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:18:57.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:18:57.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:18:57.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:18:57.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:18:57.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:18:57.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:18:57.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:18:57.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:18:57.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:18:57.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:18:57.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:18:57.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:18:57.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:18:58.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12960093 2024-09-16T16:18:58.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12960093 '!=' null 2024-09-16T16:18:58.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:18:58.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:58.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:18:58.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:18:58.557 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:58.558 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:18:58.559 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:18:58.560 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:18:58.560 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:18:58.560 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:18:58.560 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:18:58.560 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:18:58.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:18:58.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:18:58.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:18:58.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:18:58.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:18:58.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:18:58.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:18:58.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:18:58.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:18:59.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:18:59.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:18:59.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:18:59.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:18:59.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:18:59.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:18:59.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:18:59.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:18:59.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:18:59.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:18:59.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:18:59.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:18:59.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:00.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19219064 2024-09-16T16:19:00.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19219064 '!=' null 2024-09-16T16:19:00.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:00.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:00.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:00.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:00.431 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:00.431 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:00.431 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:00.431 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:00.431 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:00.431 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:00.431 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:00.431 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:00.431 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:00.431 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:00.431 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:00.432 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:00.433 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:00.433 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:00.433 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:00.433 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:00.433 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:00.433 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:00.433 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:00.433 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:00.433 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:00.433 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:00.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:00.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:00.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:00.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:00.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:00.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:00.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:00.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:00.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:00.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:00.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:00.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:00.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:01.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:01.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:01.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:01.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:01.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:01.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:01.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:01.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:01.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:02.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598145 2024-09-16T16:19:02.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598145 '!=' null 2024-09-16T16:19:02.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:02.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:02.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:02.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:02.224 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:02.224 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:02.224 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:02.224 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:02.224 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:02.224 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:02.224 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:02.224 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:02.224 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:02.224 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:02.224 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:02.225 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:02.226 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:02.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:02.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:02.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:02.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:02.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:02.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:02.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:02.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:02.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:02.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:02.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:02.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:03.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:03.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:03.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:03.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:03.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:03.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:03.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:03.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:03.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:04.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597789 2024-09-16T16:19:04.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597789 '!=' null 2024-09-16T16:19:04.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:04.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:04.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:04.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:04.212 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:04.212 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:04.213 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:04.214 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:04.215 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:04.215 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:04.215 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:04.215 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:04.215 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:04.215 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:04.215 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:04.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:04.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:04.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:04.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:04.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:04.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:04.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:04.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:04.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:04.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:04.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:04.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:04.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:05.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:05.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:05.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:05.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:05.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:05.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:05.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:05.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:05.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333721 2024-09-16T16:19:05.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333721 '!=' null 2024-09-16T16:19:05.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:05.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:05.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:05.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:06.035 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:06.035 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:06.035 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:06.035 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:06.036 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:06.037 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:06.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:06.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:06.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:06.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:06.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:06.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:06.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:06.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:06.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:06.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:06.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:06.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:06.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:07.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:07.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:07.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:07.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:07.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:07.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:07.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:07.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:07.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:07.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598067 2024-09-16T16:19:07.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598067 '!=' null 2024-09-16T16:19:07.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:07.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:07.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:07.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:07.944 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:07.944 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:07.944 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:07.944 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:07.944 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:07.944 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:07.945 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:07.946 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:07.947 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:07.947 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:07.947 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:07.947 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:07.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:08.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:08.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:08.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:08.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:08.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:08.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:08.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:08.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:08.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:08.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:08.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:08.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:09.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:09.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:09.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:09.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:09.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:09.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:09.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:09.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:09.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:09.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333033 2024-09-16T16:19:09.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333033 '!=' null 2024-09-16T16:19:09.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:09.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:09.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:09.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:09.954 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:09.954 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:09.954 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:09.954 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:09.954 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:09.954 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:09.954 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:09.954 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:09.954 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:09.955 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:09.956 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:09.957 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:09.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:10.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:10.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:10.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:10.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:10.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:10.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:10.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:10.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:10.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:10.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:10.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:10.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:11.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:11.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:11.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:11.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:11.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:11.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:11.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:11.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:11.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:11.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471027 2024-09-16T16:19:11.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471027 '!=' null 2024-09-16T16:19:11.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:11.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:11.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:11.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:11.866 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:11.868 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:11.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:11.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:11.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:11.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:11.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:11.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:11.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:11.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:11.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:12.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:12.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:12.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:12.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:13.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:13.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:13.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:13.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:13.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:13.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:13.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:13.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:13.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:13.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470929 2024-09-16T16:19:13.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470929 '!=' null 2024-09-16T16:19:13.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:13.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:13.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:13.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:13.715 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:13.715 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:13.715 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:13.715 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:13.715 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:13.716 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:13.717 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:13.718 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:13.718 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:13.718 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:13.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:13.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:13.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:13.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:13.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:13.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:13.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:13.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:13.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:14.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:14.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:14.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:14.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:14.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:14.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:14.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:14.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:14.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:14.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:14.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:14.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:14.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207309 2024-09-16T16:19:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207309 '!=' null 2024-09-16T16:19:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:15.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:15.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:15.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:15.609 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:15.609 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:15.609 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:15.609 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:15.610 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:15.611 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:15.612 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:15.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:15.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:15.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:15.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:15.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:15.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:15.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:15.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:15.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:16.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:16.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:16.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:16.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:16.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:16.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:16.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:16.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:16.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:16.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:16.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:16.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:16.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:17.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471246 2024-09-16T16:19:17.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471246 '!=' null 2024-09-16T16:19:17.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:17.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:17.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:17.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:17.477 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:17.477 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:17.477 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:17.477 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:17.477 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:17.477 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:17.477 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:17.477 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:17.478 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:17.479 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:17.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:17.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:17.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:17.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:17.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:17.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:17.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:17.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:17.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:18.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:18.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:18.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:18.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:18.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:18.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:18.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:18.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:18.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:18.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:18.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:18.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:18.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:19.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206097 2024-09-16T16:19:19.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206097 '!=' null 2024-09-16T16:19:19.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:19.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:19.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:19.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:19.353 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:19.353 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:19.353 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:19.353 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:19.353 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:19.353 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:19.353 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:19.353 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:19.353 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:19.353 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:19.353 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:19.354 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:19.355 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:19.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:19.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:19.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:19.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:19.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:19.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:19.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:19.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:19.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:19.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:19.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:19.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:19.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:20.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:20.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:20.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:20.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:20.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:20.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:20.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:20.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:20.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:21.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206097 2024-09-16T16:19:21.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206097 '!=' null 2024-09-16T16:19:21.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:21.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:21.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:21.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:21.196 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:21.196 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:21.196 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:21.197 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:21.198 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:21.199 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:21.199 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:21.199 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:21.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:21.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:21.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:21.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:21.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:21.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:21.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:21.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:21.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:21.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:21.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:21.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:21.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:22.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:22.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:22.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:22.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:22.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:22.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:22.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:22.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:22.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:22.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470869 2024-09-16T16:19:22.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470869 '!=' null 2024-09-16T16:19:22.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:22.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:22.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:22.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:23.044 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:23.044 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:23.044 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:23.044 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:23.044 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:23.044 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:23.044 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:23.044 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:23.044 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:23.045 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:23.046 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:23.046 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:23.046 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:23.046 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:23.046 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:23.046 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:23.046 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:23.046 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:23.046 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:23.046 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:23.046 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:23.047 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:23.047 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:23.047 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:23.047 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:23.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:23.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:23.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:23.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:23.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:23.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:23.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:23.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:23.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:23.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:23.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:24.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:24.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:24.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:24.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:24.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:24.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:24.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:24.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:24.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:24.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470881 2024-09-16T16:19:24.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470881 '!=' null 2024-09-16T16:19:24.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:24.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:24.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:24.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:24.915 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:24.915 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:24.915 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:24.915 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:24.915 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:24.915 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:24.915 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:24.916 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:24.916 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:24.916 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:24.916 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:24.916 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:24.916 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:24.916 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:24.916 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:24.916 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:24.917 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:24.917 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:24.917 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:24.917 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:24.917 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:24.917 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:24.917 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:24.917 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:24.917 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:24.918 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:24.918 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:24.918 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:24.918 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:24.918 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:24.918 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:24.918 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:24.918 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:24.918 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:24.918 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:24.919 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:24.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:25.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:25.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:25.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:25.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:25.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:25.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:25.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:25.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:25.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:25.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:25.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:25.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:26.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:26.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:26.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:26.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:26.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26205789 2024-09-16T16:19:26.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26205789 '!=' null 2024-09-16T16:19:26.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:26.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:26.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:26.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:26.856 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:26.857 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:26.858 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:26.859 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:26.859 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:26.859 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:26.859 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:26.859 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:26.859 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:26.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:26.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:26.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:26.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:26.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:26.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:26.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:26.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:27.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:27.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:27.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:27.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:27.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:27.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:27.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:27.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:27.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:27.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:27.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:27.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:27.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:28.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470346 2024-09-16T16:19:28.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470346 '!=' null 2024-09-16T16:19:28.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:28.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:28.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:28.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:28.664 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:28.664 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:28.664 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:28.664 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:28.664 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:28.664 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:28.664 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:28.664 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:28.664 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:28.665 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:28.666 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:28.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:28.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:28.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:28.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:28.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:28.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:28.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:28.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:28.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:29.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:29.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:29.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:29.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:29.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:29.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:29.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:29.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:29.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:29.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:29.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:30.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26205330 2024-09-16T16:19:30.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26205330 '!=' null 2024-09-16T16:19:30.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:30.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:30.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:30.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:30.617 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:30.617 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:30.617 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:30.617 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:30.617 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:30.617 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:30.618 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:30.619 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:30.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:30.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:30.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:30.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:30.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:30.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:30.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:30.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:30.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:31.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:31.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:31.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:31.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:31.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:31.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:31.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:31.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:31.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:31.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:31.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:31.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:31.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:32.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470838 2024-09-16T16:19:32.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470838 '!=' null 2024-09-16T16:19:32.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:32.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:32.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:32.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:32.586 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:32.589 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:32.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:32.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:32.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:32.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:32.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:32.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:32.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:32.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:32.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:33.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:33.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:33.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:33.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:33.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:33.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:33.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:33.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:33.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:33.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:33.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:33.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:33.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:34.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470858 2024-09-16T16:19:34.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470858 '!=' null 2024-09-16T16:19:34.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:34.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:34.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:34.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:34.544 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:34.544 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:34.544 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:34.544 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:34.544 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:34.544 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:34.545 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:34.546 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:34.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:34.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:34.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:34.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:34.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:34.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:34.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:34.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:34.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:35.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:35.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:35.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:35.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:35.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:35.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:35.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:35.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:35.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:35.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:35.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:35.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:35.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:36.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25332722 2024-09-16T16:19:36.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25332722 '!=' null 2024-09-16T16:19:36.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:36.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:36.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:36.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:36.489 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:36.489 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:36.489 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:36.489 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:36.489 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:36.489 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:36.489 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:36.489 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:36.489 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:36.489 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:36.490 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:36.491 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:36.491 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:36.491 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:36.491 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:36.491 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:36.491 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:36.491 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:36.491 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:36.491 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:36.491 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:36.492 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:36.492 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:36.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:36.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:36.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:36.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:36.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:36.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:36.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:36.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:36.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:37.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:37.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:37.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:37.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:37.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:37.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:37.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:37.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:37.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:37.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:37.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:37.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:37.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:38.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597552 2024-09-16T16:19:38.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597552 '!=' null 2024-09-16T16:19:38.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:38.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:38.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:38.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:38.269 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:38.270 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:38.271 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:38.271 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:38.271 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:38.271 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:38.271 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:38.271 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:38.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:38.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:38.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:38.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:38.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:38.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:38.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:38.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:38.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:38.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:38.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:38.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:38.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:39.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:39.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:39.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:39.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25332662 2024-09-16T16:19:39.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25332662 '!=' null 2024-09-16T16:19:39.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:39.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:39.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:39.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:40.074 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:40.075 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:40.076 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:40.076 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:40.076 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:40.076 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:40.076 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:40.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:40.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:40.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:40.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:40.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:40.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:40.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:40.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:40.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:40.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:40.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:40.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:40.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:41.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:41.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:41.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:41.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:41.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:41.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:41.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:41.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:41.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597581 2024-09-16T16:19:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597581 '!=' null 2024-09-16T16:19:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:41.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:41.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:41.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:41.908 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:41.908 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:41.908 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:41.908 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:41.908 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:41.908 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:41.908 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:41.909 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:41.909 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:41.909 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:41.909 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:41.909 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:41.909 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:41.909 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:41.909 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:41.909 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:41.909 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:41.911 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:41.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:42.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:42.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:42.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:42.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:42.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:42.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:42.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:42.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:42.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:42.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:42.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:42.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:43.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:43.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:43.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:43.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597153 2024-09-16T16:19:43.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597153 '!=' null 2024-09-16T16:19:43.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:43.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:43.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:43.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:43.856 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:43.857 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:43.857 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:43.857 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:43.857 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:43.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:43.857 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:43.857 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:43.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:43.858 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:43.858 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:43.858 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:43.858 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:43.858 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:43.858 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:43.858 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:43.858 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:43.858 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:43.858 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:43.858 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:43.859 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:43.859 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:43.859 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:43.859 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:43.859 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:43.859 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:43.859 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:43.859 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:43.859 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:43.859 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:43.859 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:43.860 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:43.860 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:43.860 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:43.860 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:43.860 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:43.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:43.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:43.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:43.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:43.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:43.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:43.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:43.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:43.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:44.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:44.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:44.475 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:44.475 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:45.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:45.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:45.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:45.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:45.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:45.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:45.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:45.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:45.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:45.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24458962 2024-09-16T16:19:45.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24458962 '!=' null 2024-09-16T16:19:45.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:45.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:45.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:45.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:46.062 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:46.062 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:46.062 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:46.062 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:46.063 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:46.064 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:46.065 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:46.065 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:46.065 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:46.065 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:46.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:46.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:46.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:46.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:46.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:46.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:46.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:46.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:46.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:46.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:46.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:46.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:46.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:47.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:47.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:47.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:47.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:47.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:47.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:47.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:47.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:47.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:47.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597153 2024-09-16T16:19:47.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597153 '!=' null 2024-09-16T16:19:47.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:47.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:47.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:47.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:47.945 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:47.946 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:47.947 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:47.947 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:47.947 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:47.947 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:47.947 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:47.947 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:47.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:48.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:48.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:48.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:48.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:48.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:48.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:48.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:48.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:48.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:48.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:48.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:48.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:49.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:49.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:49.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:49.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:49.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:49.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:49.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:49.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:49.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:49.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25332511 2024-09-16T16:19:49.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25332511 '!=' null 2024-09-16T16:19:49.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:49.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:49.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:49.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:49.875 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:49.876 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:49.876 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:49.876 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:49.876 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:49.876 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:49.876 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:49.876 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:49.876 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:49.876 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:49.877 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:49.877 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:49.877 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:49.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:49.877 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:49.877 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:49.877 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:49.877 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:49.877 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:49.877 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:49.877 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:49.878 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:49.878 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:49.878 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:49.878 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:49.878 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:49.878 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:49.878 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:49.878 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:49.878 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:49.878 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:49.878 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:49.879 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:49.879 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:49.879 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:49.879 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:49.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:49.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:49.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:49.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:49.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:49.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:49.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:49.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:49.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:50.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:50.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:50.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:50.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:51.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:51.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:51.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:51.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:51.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:51.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:51.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:51.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:51.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:51.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597415 2024-09-16T16:19:51.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597415 '!=' null 2024-09-16T16:19:51.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:51.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:51.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:51.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:51.737 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:51.737 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:51.737 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:51.737 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:51.737 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:51.737 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:51.737 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:51.737 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:51.738 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:51.739 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:51.740 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:51.740 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:51.740 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:51.740 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:51.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:51.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:51.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:51.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:51.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:51.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:51.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:51.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:51.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:52.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:52.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:52.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:52.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:52.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:52.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:52.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:52.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:52.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:52.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:52.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:52.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:52.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597525 2024-09-16T16:19:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597525 '!=' null 2024-09-16T16:19:53.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:53.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:53.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:53.697 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:53.697 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:53.697 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:53.697 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:53.697 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:53.697 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:53.697 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:53.697 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:53.697 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:53.698 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:53.698 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:53.698 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:53.698 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:53.698 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:53.698 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:53.698 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:53.698 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:53.698 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:53.698 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:53.698 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:53.699 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:53.699 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:53.699 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:53.699 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:53.699 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:53.699 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:53.699 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:53.699 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:53.699 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:53.699 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:53.699 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:53.700 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:53.700 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:53.700 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:53.700 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:53.700 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:53.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:53.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:53.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:53.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:53.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:53.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:53.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:53.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:53.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:54.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:54.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:54.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:54.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:54.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:54.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:54.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:54.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:54.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:54.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:54.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:54.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:54.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:55.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333205 2024-09-16T16:19:55.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333205 '!=' null 2024-09-16T16:19:55.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:55.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:55.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:55.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:55.721 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:55.722 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:55.722 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:55.722 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:55.722 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:55.722 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:55.722 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:55.722 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:55.722 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:55.722 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:55.722 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:55.723 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:55.723 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:55.723 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:55.723 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:55.723 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:55.723 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:55.723 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:55.723 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:55.723 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:55.723 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:55.723 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:55.724 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:55.724 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:55.724 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:55.724 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:55.724 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:55.724 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:55.724 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:55.724 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:55.724 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:55.724 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:55.725 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:55.725 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:55.725 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:55.725 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:55.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:55.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:55.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:55.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:55.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:55.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:55.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:55.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:55.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:56.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:56.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:56.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:56.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:56.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:56.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:56.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:56.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:56.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:56.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:56.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:56.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:57.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471301 2024-09-16T16:19:57.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471301 '!=' null 2024-09-16T16:19:57.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:57.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:57.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:57.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:57.487 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:57.487 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:57.487 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:57.487 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:57.487 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:57.487 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:57.487 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:57.487 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:57.488 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:57.489 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:57.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:57.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:57.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:57.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:57.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:57.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:57.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:57.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:57.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:58.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:58.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:58.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:19:58.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:58.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:19:58.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:19:58.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:19:58.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:19:58.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:19:58.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:19:58.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:19:58.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:19:58.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:19:59.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207524 2024-09-16T16:19:59.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207524 '!=' null 2024-09-16T16:19:59.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:19:59.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:59.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:19:59.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:19:59.410 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:19:59.410 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:19:59.410 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:59.410 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:59.410 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:19:59.410 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:59.410 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:19:59.410 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:19:59.410 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:59.410 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:19:59.411 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:19:59.412 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:59.412 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:19:59.412 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:19:59.412 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:19:59.412 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:19:59.412 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:19:59.412 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:19:59.412 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:19:59.412 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:19:59.412 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:19:59.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:19:59.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:19:59.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:19:59.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:19:59.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:19:59.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:19:59.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:19:59.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:19:59.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:19:59.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:19:59.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:19:59.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:19:59.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:00.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:00.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:00.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:00.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:00.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:00.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:00.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:00.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:00.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:01.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471880 2024-09-16T16:20:01.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471880 '!=' null 2024-09-16T16:20:01.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:01.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:01.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:01.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:01.454 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:01.454 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:01.454 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:01.455 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:01.456 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:01.457 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:01.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:01.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:01.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:01.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:01.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:01.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:01.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:01.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:01.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:01.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:01.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:01.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:01.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:02.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:02.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:02.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:02.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:02.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:02.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:02.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:02.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:02.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:03.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471718 2024-09-16T16:20:03.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471718 '!=' null 2024-09-16T16:20:03.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:03.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:03.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:03.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:03.458 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:03.458 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:03.458 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:03.458 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:03.458 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:03.458 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:03.458 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:03.458 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:03.458 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:03.458 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:03.458 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:03.459 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:03.460 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:03.460 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:03.460 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:03.460 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:03.460 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:03.460 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:03.460 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:03.460 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:03.460 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:03.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:03.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:03.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:03.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:03.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:03.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:03.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:03.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:03.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:04.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:04.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:04.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:04.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:04.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:04.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:04.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:04.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:04.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:04.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:04.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:04.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:04.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:05.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333383 2024-09-16T16:20:05.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333383 '!=' null 2024-09-16T16:20:05.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:05.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:05.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:05.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:05.676 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:05.676 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:05.676 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:05.676 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:05.676 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:05.676 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:05.676 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:05.676 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:05.676 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:05.676 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:05.676 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:05.677 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:05.678 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:05.678 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:05.678 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:05.678 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:05.678 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:05.678 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:05.678 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:05.678 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:05.678 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:05.678 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:05.678 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:05.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:05.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:05.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:05.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:05.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:05.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:05.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:05.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:05.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:06.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:06.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:06.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:06.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:06.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:06.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:06.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:06.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:06.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:06.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:06.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:06.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:06.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597959 2024-09-16T16:20:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597959 '!=' null 2024-09-16T16:20:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:07.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:07.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:07.483 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:07.484 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:07.485 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:07.486 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:07.486 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:07.486 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:07.486 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:07.486 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:07.486 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:07.486 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:07.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:07.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:07.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:07.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:07.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:07.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:07.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:07.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:07.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:08.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:08.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:08.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:08.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:08.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:08.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:08.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:08.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:08.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:08.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:08.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:08.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:08.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:09.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25334176 2024-09-16T16:20:09.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25334176 '!=' null 2024-09-16T16:20:09.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:09.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:09.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:09.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:09.510 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:09.510 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:09.510 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:09.511 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:09.512 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:09.513 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:09.513 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:09.513 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:09.513 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:09.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:09.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:09.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:09.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:09.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:09.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:09.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:09.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:09.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:10.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:10.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:10.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:10.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:10.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:10.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:10.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:10.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:10.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:10.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:10.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:10.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:10.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:11.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597850 2024-09-16T16:20:11.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597850 '!=' null 2024-09-16T16:20:11.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:11.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:11.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:11.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:11.491 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:11.492 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:11.493 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:11.494 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:11.494 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:11.494 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:11.494 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:11.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:11.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:11.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:11.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:11.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:11.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:11.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:11.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:11.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:12.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:12.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:12.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:12.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:12.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:12.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:12.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:12.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:12.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:12.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:12.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:12.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:12.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:13.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597484 2024-09-16T16:20:13.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597484 '!=' null 2024-09-16T16:20:13.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:13.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:13.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:13.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:13.496 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:13.496 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:13.496 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:13.496 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:13.497 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:13.498 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:13.499 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:13.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:13.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:13.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:13.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:13.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:13.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:13.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:13.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:13.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:14.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:14.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:14.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:14.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:14.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:14.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:14.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:14.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:15.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459646 2024-09-16T16:20:15.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459646 '!=' null 2024-09-16T16:20:15.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:15.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:15.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:15.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:15.569 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:15.569 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:15.570 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:15.571 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:15.572 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:15.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:15.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:15.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:15.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:15.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:15.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:15.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:15.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:15.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:16.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:16.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:16.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:16.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:16.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:16.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:16.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:16.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:16.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:16.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:16.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:16.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:16.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598101 2024-09-16T16:20:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598101 '!=' null 2024-09-16T16:20:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:17.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:17.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:17.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:17.371 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:17.371 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:17.372 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:17.373 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:17.374 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:17.374 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:17.374 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:17.374 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:17.374 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:17.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:17.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:17.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:17.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:17.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:17.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:17.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:17.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:17.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:17.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:17.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:17.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:17.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:18.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:18.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:18.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:18.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:18.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:18.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:18.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:18.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:18.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:19.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598101 2024-09-16T16:20:19.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598101 '!=' null 2024-09-16T16:20:19.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:19.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:19.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:19.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:19.369 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:19.369 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:19.370 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:19.371 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:19.372 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:19.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:19.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:19.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:19.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:19.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:19.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:19.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:19.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:19.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:20.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:20.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:20.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:20.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:20.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:20.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:20.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:20.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:20.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:20.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:20.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:20.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:20.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:21.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333447 2024-09-16T16:20:21.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333447 '!=' null 2024-09-16T16:20:21.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:21.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:21.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:21.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:21.371 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:21.371 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:21.371 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:21.371 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:21.371 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:21.371 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:21.372 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:21.373 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:21.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:21.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:21.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:21.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:21.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:21.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:21.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:21.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:21.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:21.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:21.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:21.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:22.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:22.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:22.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:22.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:22.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:22.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:22.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:22.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:22.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:23.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597315 2024-09-16T16:20:23.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597315 '!=' null 2024-09-16T16:20:23.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:23.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:23.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:23.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:23.228 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:23.228 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:23.228 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:23.228 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:23.228 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:23.228 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:23.228 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:23.228 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:23.229 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:23.230 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:23.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:23.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:23.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:23.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:23.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:23.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:23.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:23.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:23.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:23.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:23.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:23.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:23.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:24.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:24.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:24.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:24.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:24.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:24.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:24.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:24.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:24.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:24.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597373 2024-09-16T16:20:24.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597373 '!=' null 2024-09-16T16:20:24.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:24.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:24.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:24.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:24.995 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:24.996 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:24.997 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:24.997 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:24.997 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:24.997 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:24.997 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:24.997 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:24.997 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:24.997 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:25.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:25.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:25.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:25.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:25.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:25.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:25.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:25.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:25.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:25.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:25.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:25.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:25.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:26.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:26.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:26.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:26.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:26.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:26.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:26.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:26.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:26.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:26.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333202 2024-09-16T16:20:26.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333202 '!=' null 2024-09-16T16:20:26.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:26.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:26.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:26.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:26.734 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:26.734 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:26.734 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:26.734 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:26.734 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:26.734 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:26.735 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:26.736 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:26.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:26.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:26.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:26.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:26.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:26.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:26.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:26.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:26.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:27.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:27.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:27.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:27.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:27.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:27.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:28.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471311 2024-09-16T16:20:28.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471311 '!=' null 2024-09-16T16:20:28.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:28.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:28.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:28.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:28.588 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:28.588 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:28.588 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:28.588 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:28.588 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:28.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:28.588 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:28.589 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "7.0", 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 255, 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:28.590 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:28.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:28.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:28.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:28.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:28.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:28.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:28.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:28.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:28.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:29.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:29.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:29.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:29.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:29.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:29.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:29.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:29.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:29.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:29.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:29.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:29.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:29.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:30.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25332592 2024-09-16T16:20:30.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25332592 '!=' null 2024-09-16T16:20:30.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:30.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:30.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:30.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:30.573 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:30.574 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:30.575 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:30.576 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:30.576 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:30.576 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:30.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:30.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:30.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:30.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:30.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:30.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:30.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:30.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:30.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:31.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:31.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:31.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:31.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:32.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:32.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:32.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:32.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:32.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:32.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:32.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:32.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:32.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:32.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24895936 2024-09-16T16:20:32.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24895936 '!=' null 2024-09-16T16:20:32.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:32.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:32.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:32.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:32.876 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:32.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:32.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:32.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:32.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:32.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:32.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:32.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:32.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:32.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:33.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:20:33.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:33.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:33.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:34.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-16T16:20:34.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:20:34.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:34.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:34.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:34.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:34.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:34.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:34.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:34.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11792638 2024-09-16T16:20:34.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11792638 '!=' null 2024-09-16T16:20:34.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:34.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:34.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:34.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:34.863 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:34.863 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:34.863 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:34.863 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:34.863 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:34.864 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:34.865 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:34.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:34.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:34.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:34.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:34.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:34.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:34.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:34.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:34.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:35.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:35.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:35.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:35.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:36.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:36.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-09-16T16:20:36.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T16:20:36.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-09-16T16:20:36.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:36.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:36.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:36.272 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:36.272 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:36.272 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:36.273 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:36.274 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:36.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:36.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:36.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:36.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:36.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:36.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:36.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:36.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:36.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:36.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:36.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:36.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:36.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:37.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:37.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:37.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:37.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:37.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:37.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:37.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:37.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:37.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:38.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11793092 2024-09-16T16:20:38.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11793092 '!=' null 2024-09-16T16:20:38.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:38.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:38.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:38.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:38.182 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:38.182 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:38.184 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:38.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:38.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:38.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:38.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:38.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:38.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:38.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:38.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:38.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:38.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:38.717 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:38.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:38.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:39.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:39.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:39.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:39.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:39.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:39.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:39.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:39.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:39.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:40.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21545326 2024-09-16T16:20:40.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21545326 '!=' null 2024-09-16T16:20:40.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:40.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:40.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:40.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:40.142 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:40.142 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:40.142 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:40.142 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:40.143 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:40.143 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:40.143 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:40.143 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:40.143 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:40.143 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:40.143 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:40.143 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:40.143 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:40.143 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:40.143 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:40.144 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:40.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:40.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:40.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:40.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:40.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:40.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:40.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:40.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:40.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:40.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:40.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:40.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:40.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:41.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:41.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:41.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:41.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:41.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:41.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:41.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:41.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:41.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:41.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19578012 2024-09-16T16:20:41.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19578012 '!=' null 2024-09-16T16:20:41.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:41.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:41.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:41.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:41.970 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:41.971 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:41.972 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:41.973 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:41.973 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:41.973 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:41.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:42.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:42.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:42.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:42.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:42.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:42.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:42.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:42.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:42.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:42.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:42.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:42.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:43.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:43.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:43.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:43.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:43.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:43.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:43.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:43.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:43.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:43.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597743 2024-09-16T16:20:43.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597743 '!=' null 2024-09-16T16:20:43.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:43.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:43.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:43.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:43.905 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:43.906 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:43.907 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:43.908 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:43.908 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:43.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:44.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:44.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:44.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:44.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:44.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:44.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:44.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:44.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:44.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:44.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:44.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:44.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:44.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:44.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:44.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:44.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:44.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:44.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:44.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:44.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:44.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:45.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459969 2024-09-16T16:20:45.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459969 '!=' null 2024-09-16T16:20:45.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:45.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:45.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:45.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:45.884 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:45.884 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:45.884 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:45.884 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:45.884 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:45.884 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:45.884 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:45.885 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:45.886 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:45.886 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:45.886 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:45.886 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:45.886 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:45.886 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:45.886 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:45.886 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:45.886 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:45.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:45.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:45.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:46.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:46.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:46.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:46.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:46.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:46.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:46.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:46.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:46.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:46.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:46.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:46.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:46.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:46.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:46.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:46.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:46.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:46.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:46.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:47.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597477 2024-09-16T16:20:47.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597477 '!=' null 2024-09-16T16:20:47.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:47.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:47.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:47.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:47.647 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:47.647 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:47.647 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:47.647 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:47.647 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:47.647 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:47.647 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:47.648 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:47.649 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:47.649 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:47.649 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:47.649 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:47.649 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:47.649 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:47.649 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:47.649 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:47.649 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:47.649 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:47.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:47.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:47.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:47.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:47.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:47.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:47.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:47.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:47.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:48.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:48.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:48.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:48.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:48.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:48.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:48.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:48.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:48.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:48.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:48.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:48.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:48.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:49.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333016 2024-09-16T16:20:49.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333016 '!=' null 2024-09-16T16:20:49.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:49.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:49.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:49.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:49.601 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:49.601 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:49.601 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:49.601 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:49.601 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:49.601 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:49.601 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:49.601 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:49.601 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:49.601 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:49.601 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:49.602 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:49.602 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:49.602 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:49.602 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:49.602 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:49.602 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:49.602 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:49.602 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:49.602 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:49.602 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:49.602 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:49.603 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:49.603 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:49.603 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:49.603 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:49.603 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:49.603 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:49.603 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:49.603 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:49.603 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:49.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:49.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:49.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:49.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:49.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:49.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:49.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:49.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:49.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:50.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:50.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:50.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:50.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:50.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:50.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:50.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:50.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:50.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:50.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:50.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:50.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:50.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:51.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597425 2024-09-16T16:20:51.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597425 '!=' null 2024-09-16T16:20:51.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:51.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:51.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:51.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:51.445 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:51.445 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:51.445 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:51.445 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:51.445 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:51.445 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:51.446 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:51.447 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:51.447 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:51.447 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:51.447 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:51.447 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:51.447 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:51.447 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:51.447 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:51.447 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:51.447 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:51.447 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:51.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:51.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:51.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:51.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:51.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:51.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:51.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:51.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:51.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:51.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:51.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:51.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:51.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:52.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:52.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:52.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:52.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:52.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:52.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:52.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:52.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:52.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597679 2024-09-16T16:20:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597679 '!=' null 2024-09-16T16:20:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:53.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:53.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:53.465 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:53.465 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:53.465 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:53.465 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:53.465 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:53.465 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:53.466 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:53.467 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:53.467 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:53.467 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:53.467 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:53.467 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:53.467 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:53.467 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:53.467 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:53.467 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:53.467 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:53.467 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:53.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:53.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:53.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:53.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:53.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:53.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:53.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:53.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:53.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:53.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:53.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:53.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:53.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:54.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:54.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:54.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:54.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:54.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:54.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:54.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:54.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:54.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:55.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597679 2024-09-16T16:20:55.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597679 '!=' null 2024-09-16T16:20:55.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:55.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:55.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:55.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:55.367 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:55.367 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:55.367 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:55.367 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:55.367 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:55.367 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:55.367 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:55.367 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:55.367 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:55.367 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:55.367 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:55.368 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:55.369 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:55.369 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:55.369 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:55.369 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:55.369 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:55.369 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:55.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:55.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:55.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:55.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:55.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:55.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:55.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:55.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:55.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:55.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:55.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:55.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:55.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:56.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:56.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:56.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:56.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:56.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:56.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:56.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:56.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:56.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:57.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597521 2024-09-16T16:20:57.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597521 '!=' null 2024-09-16T16:20:57.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:57.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:57.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:57.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:57.390 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:57.390 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:57.390 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:57.390 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:57.390 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:57.390 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:57.390 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:57.390 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:57.391 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:57.392 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:57.392 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:57.392 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:57.392 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:57.392 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:57.392 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:57.392 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:57.392 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:57.392 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:57.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:57.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:57.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:57.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:57.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:57.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:57.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:57.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:57.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:57.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:57.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:57.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:57.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:20:58.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:20:58.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:20:58.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:20:58.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:20:58.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:20:58.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:20:58.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:20:58.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:20:58.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:20:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597521 2024-09-16T16:20:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597521 '!=' null 2024-09-16T16:20:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:20:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:59.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:20:59.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:20:59.243 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:20:59.243 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:20:59.243 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:59.243 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:59.243 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:20:59.243 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:59.243 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:20:59.243 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:20:59.243 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:59.244 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:20:59.245 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:20:59.245 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:20:59.245 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:20:59.245 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:20:59.245 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:20:59.245 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:20:59.245 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:20:59.245 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:20:59.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:20:59.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:20:59.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:20:59.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:20:59.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:20:59.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:20:59.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:20:59.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:20:59.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:20:59.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:20:59.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:20:59.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:20:59.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:00.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:00.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:00.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:00.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:00.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:00.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:00.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:00.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:00.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:01.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333180 2024-09-16T16:21:01.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333180 '!=' null 2024-09-16T16:21:01.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:01.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:01.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:01.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:01.149 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:01.149 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:01.149 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:01.149 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:01.149 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:01.150 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:01.151 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:01.151 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:01.151 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:01.151 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:01.151 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:01.151 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:01.151 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:01.151 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:01.151 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:01.151 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:01.151 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:01.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:01.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:01.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:01.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:01.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:01.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:01.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:01.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:01.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:01.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:01.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:01.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:01.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:02.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:02.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:02.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597480 2024-09-16T16:21:02.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597480 '!=' null 2024-09-16T16:21:02.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:02.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:02.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:02.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:02.961 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:02.961 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:02.961 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:02.961 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:02.961 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:02.961 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:02.961 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:02.962 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:02.963 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:02.963 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:02.963 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:02.963 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:02.963 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:02.963 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:02.963 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:02.963 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:02.963 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:02.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:03.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:03.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:03.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:03.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:03.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:03.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:03.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:03.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:03.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:03.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:03.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:03.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:04.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:04.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:04.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:04.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:04.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:04.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:04.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:04.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:04.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:04.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597316 2024-09-16T16:21:04.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597316 '!=' null 2024-09-16T16:21:04.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:04.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:04.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:04.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:04.849 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:04.849 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:04.849 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:04.849 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:04.849 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:04.849 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:04.849 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:04.849 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:04.849 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:04.850 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:04.851 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:04.851 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:04.851 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:04.851 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:04.851 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:04.851 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:04.851 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:04.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:04.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:04.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:04.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:04.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:04.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:04.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:04.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:04.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:05.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:05.392 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:05.392 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:05.392 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:06.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:06.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:06.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:06.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:06.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:06.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:06.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:06.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:06.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:06.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24458862 2024-09-16T16:21:06.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24458862 '!=' null 2024-09-16T16:21:06.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:06.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:06.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:06.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:06.855 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:06.855 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:06.856 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:06.857 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:06.858 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:06.858 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:06.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:06.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:06.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:06.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:06.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:06.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:06.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:06.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:06.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:07.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:07.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:07.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:07.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:08.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:08.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:08.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:08.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:08.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:08.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:08.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:08.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:08.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:08.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597006 2024-09-16T16:21:08.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597006 '!=' null 2024-09-16T16:21:08.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:08.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:08.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:08.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:08.801 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:08.801 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:08.801 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:08.801 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:08.801 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:08.801 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:08.801 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:08.802 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:08.803 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:08.803 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:08.803 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:08.803 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:08.803 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:08.803 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:08.803 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:08.803 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:08.803 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:08.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:08.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:08.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:08.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:08.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:08.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:08.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:08.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:08.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:09.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:09.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:09.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:09.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:09.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:09.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:10.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459067 2024-09-16T16:21:10.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459067 '!=' null 2024-09-16T16:21:10.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:10.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:10.717 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:10.717 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:10.831 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:10.831 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:10.832 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:10.833 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:10.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:10.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:10.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:10.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:10.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:10.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:10.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:10.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:10.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:11.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:11.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:11.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:11.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:12.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:12.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:12.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:12.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:12.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:12.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:12.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:12.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:12.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:12.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723653 2024-09-16T16:21:12.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723653 '!=' null 2024-09-16T16:21:12.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:12.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:12.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:12.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:12.762 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:12.763 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:12.765 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:12.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:12.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:12.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:12.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:12.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:12.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:12.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:12.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:12.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:13.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:13.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:13.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:13.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:14.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:14.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:14.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:14.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:14.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:14.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:14.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:14.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:14.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:14.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723485 2024-09-16T16:21:14.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723485 '!=' null 2024-09-16T16:21:14.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:14.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:14.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:14.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:14.672 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:14.672 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:14.672 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:14.672 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:14.672 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:14.672 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:14.672 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:14.673 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:14.674 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:14.674 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:14.674 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:14.674 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:14.674 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:14.674 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:14.674 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:14.674 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:14.674 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:14.674 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:14.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:14.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:14.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:14.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:14.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:14.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:14.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:14.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:14.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:15.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:15.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:15.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:15.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:15.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:15.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:15.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:15.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:15.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:15.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:15.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:15.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:15.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:16.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585770 2024-09-16T16:21:16.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585770 '!=' null 2024-09-16T16:21:16.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:16.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:16.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:16.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:16.824 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:16.824 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:16.824 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:16.824 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:16.824 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:16.824 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:16.824 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:16.825 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:16.826 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:16.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:16.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:16.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:16.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:16.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:16.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:16.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:16.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:16.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:17.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:17.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:17.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:17.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:17.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:17.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:17.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:17.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:17.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:17.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:17.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:17.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:17.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:18.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724116 2024-09-16T16:21:18.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724116 '!=' null 2024-09-16T16:21:18.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:18.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:18.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:18.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:18.770 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:18.772 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:18.773 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:18.773 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:18.773 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:18.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:18.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:18.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:18.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:18.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:18.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:18.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:18.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:18.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:19.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:19.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:19.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:19.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:20.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:20.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:20.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:20.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:20.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:20.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:20.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:20.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:20.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:20.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459903 2024-09-16T16:21:20.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459903 '!=' null 2024-09-16T16:21:20.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:20.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:20.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:20.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:20.900 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:20.901 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:20.902 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:20.903 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:20.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:21.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:21.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:21.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:21.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:21.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:21.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:21.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:21.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:21.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:21.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:21.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:21.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:22.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:22.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:22.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:22.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:22.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:22.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:22.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:22.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:22.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:22.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597873 2024-09-16T16:21:22.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597873 '!=' null 2024-09-16T16:21:22.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:22.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:22.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:22.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:22.752 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:22.752 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:22.752 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:22.752 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:22.752 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:22.752 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:22.752 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:22.753 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:22.754 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:22.754 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:22.754 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:22.754 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:22.754 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:22.754 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:22.754 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:22.754 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:22.754 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:22.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:22.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:22.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:22.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:22.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:22.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:22.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:22.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:22.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:23.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:23.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:23.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:23.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:23.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:23.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:23.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:23.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:23.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:23.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:23.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:23.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:23.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:24.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598183 2024-09-16T16:21:24.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598183 '!=' null 2024-09-16T16:21:24.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:24.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:24.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:24.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:24.579 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:24.579 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:24.579 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:24.579 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:24.579 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:24.579 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:24.579 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:24.579 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:24.579 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:24.579 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:24.579 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:24.580 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:24.581 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:24.581 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:24.581 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:24.581 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:24.581 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:24.581 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:24.581 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:24.581 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:24.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:24.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:24.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:24.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:24.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:24.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:24.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:24.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:24.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:25.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:25.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:25.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:25.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:25.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:25.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:25.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:25.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:25.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:25.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:25.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:25.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:25.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24461018 2024-09-16T16:21:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24461018 '!=' null 2024-09-16T16:21:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:26.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:26.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:26.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:26.478 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:26.478 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:26.478 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:26.478 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:26.478 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:26.478 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:26.478 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:26.478 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:26.478 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:26.478 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:26.479 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:26.480 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:26.480 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:26.480 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:26.480 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:26.480 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:26.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:26.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:26.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:26.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:26.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:26.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:26.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:26.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:26.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:27.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:27.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:27.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:27.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:27.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:27.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:27.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:27.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:27.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:27.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:27.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:27.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:27.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:28.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597946 2024-09-16T16:21:28.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597946 '!=' null 2024-09-16T16:21:28.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:28.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:28.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:28.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:28.232 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:28.232 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:28.232 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:28.232 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:28.232 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:28.232 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:28.232 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:28.232 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:28.232 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:28.233 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:28.234 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:28.234 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:28.234 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:28.234 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:28.234 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:28.234 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:28.234 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:28.234 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:28.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:28.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:28.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:28.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:28.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:28.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:28.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:28.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:28.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:28.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:28.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:28.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:28.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:29.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:29.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:29.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:29.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:29.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:29.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:29.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:29.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:29.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:29.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459760 2024-09-16T16:21:29.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459760 '!=' null 2024-09-16T16:21:29.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:29.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:29.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:29.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:30.076 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:30.077 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:30.078 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:30.078 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:30.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:30.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:30.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:30.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:30.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:30.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:30.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:30.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:30.193 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:30.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:30.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:30.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:30.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:31.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:31.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:31.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:31.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:31.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:31.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:31.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:31.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:31.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:31.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724062 2024-09-16T16:21:31.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724062 '!=' null 2024-09-16T16:21:31.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:31.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:31.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:31.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:31.815 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:31.815 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:31.815 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:31.815 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:31.815 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:31.815 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:31.815 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:31.815 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:31.816 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:31.817 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:31.817 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:31.817 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:31.817 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:31.817 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:31.817 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:31.817 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:31.817 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:31.817 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:31.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:31.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:31.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:31.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:31.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:31.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:31.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:31.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:31.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:32.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:32.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:32.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:32.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:32.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:32.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:32.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:32.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:32.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:32.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:32.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:32.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:32.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:33.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723956 2024-09-16T16:21:33.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723956 '!=' null 2024-09-16T16:21:33.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:33.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:33.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:33.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:33.940 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:33.942 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:33.943 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:33.943 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:33.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:34.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:34.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:34.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:34.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:34.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:34.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:34.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:34.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:34.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:34.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:34.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:34.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:35.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:35.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:35.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:35.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:35.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:35.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:35.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:35.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:35.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:35.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585647 2024-09-16T16:21:35.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585647 '!=' null 2024-09-16T16:21:35.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:35.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:35.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:35.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:35.711 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:35.789 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:35.789 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:35.789 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:35.789 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:35.789 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:35.789 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:35.790 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:35.791 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:35.791 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:35.791 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:35.791 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:35.791 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:35.791 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:35.791 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:35.791 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:35.791 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:35.791 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:35.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:35.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:35.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:35.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:35.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:35.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:35.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:35.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:35.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:36.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:36.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:36.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:36.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:36.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:36.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:36.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:36.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:36.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:36.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:36.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:36.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:36.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:37.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723600 2024-09-16T16:21:37.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723600 '!=' null 2024-09-16T16:21:37.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:37.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:37.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:37.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:37.688 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:37.688 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:37.688 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:37.688 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:37.688 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:37.688 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:37.688 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:37.688 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:37.688 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:37.688 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:37.688 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:37.689 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:37.690 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:37.690 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:37.690 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:37.690 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:37.690 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:37.690 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:37.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:37.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:37.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:37.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:37.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:37.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:37.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:37.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:37.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:38.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:38.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:38.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:38.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:38.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:38.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:38.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:38.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:38.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:38.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:39.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585820 2024-09-16T16:21:39.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585820 '!=' null 2024-09-16T16:21:39.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:39.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:39.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:39.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:39.512 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:39.512 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:39.512 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:39.512 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:39.512 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:39.513 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:39.514 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:39.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:39.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:39.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:39.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:39.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:39.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:39.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:39.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:39.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:40.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:40.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:40.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:40.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:40.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:40.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:40.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:40.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:40.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:40.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:40.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:40.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:40.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:41.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724089 2024-09-16T16:21:41.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724089 '!=' null 2024-09-16T16:21:41.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:41.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:41.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:41.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:41.384 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:41.384 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:41.384 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:41.384 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:41.384 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:41.384 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:41.384 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:41.384 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:41.384 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:41.384 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:41.385 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:41.386 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:41.386 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:41.386 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:41.386 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:41.386 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:41.386 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:41.386 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:41.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:41.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:41.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:41.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:41.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:41.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:41.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:41.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:41.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:41.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:41.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:41.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:41.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:42.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:42.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:42.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:42.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:42.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:42.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:42.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:42.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:42.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:43.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724089 2024-09-16T16:21:43.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724089 '!=' null 2024-09-16T16:21:43.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:43.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:43.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:43.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:43.279 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:43.280 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:43.281 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:43.282 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:43.282 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:43.282 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:43.282 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:43.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:43.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:43.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:43.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:43.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:43.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:43.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:43.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:43.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:43.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:43.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:43.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:43.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:44.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:44.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:44.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:44.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:44.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:44.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:44.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:44.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:44.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:45.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724122 2024-09-16T16:21:45.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724122 '!=' null 2024-09-16T16:21:45.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:45.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:45.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:45.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:45.262 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:45.263 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:45.264 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:45.264 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:45.264 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:45.264 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:45.264 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:45.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:45.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:45.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:45.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:45.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:45.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:45.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:45.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:45.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:45.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:45.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:45.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:45.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:46.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:46.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:46.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:46.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:46.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:46.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:46.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:46.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:46.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:47.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597232 2024-09-16T16:21:47.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597232 '!=' null 2024-09-16T16:21:47.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:47.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:47.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:47.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:47.394 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:47.394 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:47.394 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:47.394 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:47.394 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:47.394 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:47.394 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:47.394 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:47.394 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:47.395 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:47.396 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:47.396 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:47.396 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:47.396 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:47.396 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:47.396 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:47.396 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:47.396 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:47.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:47.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:47.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:47.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:47.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:47.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:47.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:47.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:47.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:47.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:47.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:47.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:47.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:48.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:48.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:48.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:48.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:48.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:48.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:48.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:48.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:48.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:49.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459137 2024-09-16T16:21:49.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459137 '!=' null 2024-09-16T16:21:49.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:49.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:49.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:49.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:49.280 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:49.280 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:49.280 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:49.280 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:49.280 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:49.280 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:49.280 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:49.280 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:49.280 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:49.280 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:49.281 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:49.282 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:49.282 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:49.282 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:49.282 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:49.282 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:49.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:49.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:49.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:49.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:49.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:49.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:49.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:49.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:49.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:49.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:49.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:49.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:49.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:50.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:50.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:50.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:50.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:50.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:50.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:50.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:50.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:50.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:51.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459137 2024-09-16T16:21:51.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459137 '!=' null 2024-09-16T16:21:51.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:51.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:51.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:51.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:51.272 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:51.272 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:51.272 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:51.273 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:51.274 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:51.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:51.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:51.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:51.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:51.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:51.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:51.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:51.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:51.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:51.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:51.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:51.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:51.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:52.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:52.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:52.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:52.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:52.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:52.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:52.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:52.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:52.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:52.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16596900 2024-09-16T16:21:52.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16596900 '!=' null 2024-09-16T16:21:52.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:52.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:52.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:52.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:53.066 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:53.066 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:53.067 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:53.068 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:53.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:53.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:53.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:53.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:53.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:53.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:53.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:53.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:53.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:53.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:53.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:53.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:54.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:54.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:54.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:54.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:54.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:54.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:54.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:54.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:54.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:54.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16596727 2024-09-16T16:21:54.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16596727 '!=' null 2024-09-16T16:21:54.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:54.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:54.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:54.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:54.955 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:54.955 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:54.955 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:54.955 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:54.955 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:54.955 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:54.955 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:54.955 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:54.955 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:54.955 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:54.956 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:54.957 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:54.957 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:54.957 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:54.957 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:54.957 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:54.957 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:54.957 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:54.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:55.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:55.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:55.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:55.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:55.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:55.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:55.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:55.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:55.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:55.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:55.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:55.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:56.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:56.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:56.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:56.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:56.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:56.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:56.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:56.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:56.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:56.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24458815 2024-09-16T16:21:56.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24458815 '!=' null 2024-09-16T16:21:56.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:56.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:56.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:56.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:56.739 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:56.739 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:56.739 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:56.739 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:56.739 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:56.740 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:56.741 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:56.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:56.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:56.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:56.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:56.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:56.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:56.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:56.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:56.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:57.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:57.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:57.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:57.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:57.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:57.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:57.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:57.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:57.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:57.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:57.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:57.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:57.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:21:58.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723747 2024-09-16T16:21:58.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723747 '!=' null 2024-09-16T16:21:58.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:21:58.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:58.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:21:58.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:21:58.618 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:21:58.618 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:21:58.618 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:58.618 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:58.618 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:21:58.618 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:58.618 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:21:58.618 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:21:58.618 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:58.618 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:21:58.619 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:21:58.620 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:21:58.620 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:21:58.620 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:21:58.620 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:21:58.620 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:21:58.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:21:58.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:21:58.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:21:58.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:21:58.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:21:58.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:21:58.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:21:58.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:21:58.733 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:21:59.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:21:59.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:21:59.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:21:59.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:21:59.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:21:59.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:21:59.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:21:59.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:21:59.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:21:59.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:21:59.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:21:59.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:21:59.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:00.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585496 2024-09-16T16:22:00.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585496 '!=' null 2024-09-16T16:22:00.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:00.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:00.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:00.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:00.583 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:00.583 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:00.583 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:22:00.584 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:00.585 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:00.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:00.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:00.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:00.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:00.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:00.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:00.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:00.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:00.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:01.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:01.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:01.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:01.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:01.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:01.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:01.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:01.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:01.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:01.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:01.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:01.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:01.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:02.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723613 2024-09-16T16:22:02.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723613 '!=' null 2024-09-16T16:22:02.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:02.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:02.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:02.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:02.561 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:02.561 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:02.561 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:02.561 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:02.561 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:22:02.561 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:02.561 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:22:02.561 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:02.562 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:02.563 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:02.563 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:02.563 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:02.563 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:02.563 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:02.563 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:02.563 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:02.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:02.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:02.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:02.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:02.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:02.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:02.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:03.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:03.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:03.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:03.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:03.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:03.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:03.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:03.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:03.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:03.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:03.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:03.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:03.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:04.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723750 2024-09-16T16:22:04.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723750 '!=' null 2024-09-16T16:22:04.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:04.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:04.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:04.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:04.496 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:04.497 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:04.498 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:04.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:04.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:04.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:04.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:04.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:04.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:04.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:04.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:04.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:05.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:05.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:05.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:05.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:05.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:05.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:05.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:05.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:05.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:05.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:05.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:05.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:05.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:06.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459112 2024-09-16T16:22:06.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459112 '!=' null 2024-09-16T16:22:06.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:06.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:06.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:06.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:06.380 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:06.381 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:06.381 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:06.381 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:06.381 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:22:06.381 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:06.381 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:22:06.381 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:22:06.381 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:06.381 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:06.381 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:06.381 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:06.382 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:06.383 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:06.383 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:06.383 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:06.383 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:06.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:06.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:06.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:06.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:06.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:06.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:06.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:06.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:06.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:06.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:06.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:06.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:06.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:07.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:07.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:07.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:07.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:07.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:07.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:07.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:07.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:07.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:08.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16596920 2024-09-16T16:22:08.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16596920 '!=' null 2024-09-16T16:22:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:08.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:08.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:08.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:08.231 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:08.231 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:08.231 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:08.232 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:08.233 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:08.234 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:08.234 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:08.234 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:08.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:08.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:08.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:08.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:08.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:08.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:08.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:08.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:08.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:08.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:08.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:08.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:08.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:09.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:09.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:09.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:09.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:09.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:09.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:09.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:09.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:09.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:09.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24458687 2024-09-16T16:22:09.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24458687 '!=' null 2024-09-16T16:22:09.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:09.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:09.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:09.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:10.029 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:10.029 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:10.030 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:10.031 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:10.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:10.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:10.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:10.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:10.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:10.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:10.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:10.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:10.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:10.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:10.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:10.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:10.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:11.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:11.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:11.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:11.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:11.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:11.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:11.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:11.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:11.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:11.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597446 2024-09-16T16:22:11.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597446 '!=' null 2024-09-16T16:22:11.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:11.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:11.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:11.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:11.887 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:11.890 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:11.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:11.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:11.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:11.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:11.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:11.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:11.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:11.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:11.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:12.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:12.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:12.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:12.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:13.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:13.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:13.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:13.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:13.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597446 2024-09-16T16:22:13.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597446 '!=' null 2024-09-16T16:22:13.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:13.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:13.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:13.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:13.760 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:13.760 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:13.760 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:13.761 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:13.761 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:22:13.761 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:13.761 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:22:13.761 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:22:13.761 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:13.761 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:13.761 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:13.761 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:13.761 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:13.762 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:13.762 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:13.762 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:22:13.762 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:13.762 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:13.762 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:22:13.762 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:13.762 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:13.762 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:13.762 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:13.762 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:13.763 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:13.763 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:13.763 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:13.763 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:13.763 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:13.763 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:13.763 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:13.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:13.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:13.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:13.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:13.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:13.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:13.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:13.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:13.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:14.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:14.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:14.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:14.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:14.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:14.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:14.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:14.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:14.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:14.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:14.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:14.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:14.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:15.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459761 2024-09-16T16:22:15.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459761 '!=' null 2024-09-16T16:22:15.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:15.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:15.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:15.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:15.676 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:15.676 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:15.676 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:15.676 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:15.676 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [ 2024-09-16T16:22:15.676 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:15.676 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 190, 2024-09-16T16:22:15.676 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "1.0", 2024-09-16T16:22:15.677 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:15.678 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:15.678 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:15.678 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:15.678 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:15.678 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:15.678 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:15.678 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:15.678 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:15.678 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:15.678 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:15.678 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:15.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:15.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:15.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:15.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:15.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:15.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:15.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:15.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:15.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:16.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:16.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:16.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:16.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:16.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:16.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:16.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:16.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:16.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:16.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:16.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:16.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:16.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:17.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724285 2024-09-16T16:22:17.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724285 '!=' null 2024-09-16T16:22:17.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:17.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:17.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:17.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:17.747 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:17.747 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:17.747 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:17.747 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:17.748 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:17.749 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:17.749 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:17.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:17.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:17.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:17.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:17.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:17.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:17.862 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:17.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:17.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:18.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:18.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:18.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:18.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:18.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:18.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:18.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:18.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:18.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:18.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:18.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:18.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:18.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:19.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586231 2024-09-16T16:22:19.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586231 '!=' null 2024-09-16T16:22:19.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:19.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:19.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:19.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:19.767 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:19.767 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:19.767 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:19.767 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:19.767 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:19.768 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:19.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:19.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:19.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:19.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:19.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:19.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:19.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:19.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:19.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:20.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:20.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:20.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:20.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:20.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:20.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:20.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:20.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:20.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:20.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:20.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:20.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:20.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:21.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586907 2024-09-16T16:22:21.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586907 '!=' null 2024-09-16T16:22:21.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:21.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:21.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:21.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:21.645 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:21.645 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:21.645 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:21.645 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:21.645 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:21.645 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:21.645 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:21.645 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:21.645 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:21.645 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:21.645 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:21.646 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:21.646 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:21.646 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:21.646 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:21.646 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:21.646 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:21.646 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:21.646 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:21.646 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:21.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:21.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:21.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:21.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:21.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:21.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:21.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:21.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:21.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:22.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:22.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:22.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:22.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:22.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:22.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:22.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:22.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:22.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:22.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:22.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:22.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:22.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:23.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11793841 2024-09-16T16:22:23.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11793841 '!=' null 2024-09-16T16:22:23.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:23.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:23.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:23.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:23.531 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:23.532 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:23.532 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:23.532 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:23.532 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:23.532 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:23.532 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:23.532 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:23.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:23.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:23.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:23.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:23.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:23.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:23.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:23.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:24.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:22:24.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:24.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:24.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:24.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-16T16:22:24.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-16T16:22:24.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:24.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:24.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:24.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:24.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:24.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:24.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:25.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11793841 2024-09-16T16:22:25.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11793841 '!=' null 2024-09-16T16:22:25.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:25.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:25.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:25.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:25.293 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:25.293 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:25.293 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:25.293 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:25.293 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:25.293 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:25.294 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:25.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:25.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:25.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:25.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:25.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:25.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:25.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:25.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:25.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:25.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:25.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:25.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:25.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:26.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:26.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 1 2024-09-16T16:22:26.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T16:22:26.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-09-16T16:22:26.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:26.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:26.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:26.474 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:26.474 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:26.474 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:26.474 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:26.474 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:26.474 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:26.474 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:26.474 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:26.474 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:26.474 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:26.475 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:26.475 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:26.475 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:26.475 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:26.475 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:26.475 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:26.475 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:26.475 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:26.475 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:26.475 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:26.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:26.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:26.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:26.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:26.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:26.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:26.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:26.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:26.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:27.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:27.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:27.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:27.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:27.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:27.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:27.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:27.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:27.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:27.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:27.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:27.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:27.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:28.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19495247 2024-09-16T16:22:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19495247 '!=' null 2024-09-16T16:22:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:28.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:28.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:28.413 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:28.413 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:28.413 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:28.413 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:28.413 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:28.414 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:28.414 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:28.414 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:28.414 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:28.414 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:28.414 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:28.414 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:28.414 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:28.414 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:28.414 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:28.415 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:28.415 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:28.415 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:28.415 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:28.415 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:28.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:28.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:28.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:28.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:28.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:28.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:28.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:28.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:28.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:28.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:28.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:28.941 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:28.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:29.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:29.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:29.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:29.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:29.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:29.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:29.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:29.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:29.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:30.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26948467 2024-09-16T16:22:30.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26948467 '!=' null 2024-09-16T16:22:30.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:30.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:30.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:30.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:30.227 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:30.227 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:30.227 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:30.227 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:30.227 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:30.227 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:30.227 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:30.227 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:30.228 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:30.228 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:30.228 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:30.228 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:30.228 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:30.228 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:30.228 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:30.228 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:30.228 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:30.228 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:30.229 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:30.229 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:30.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:30.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:30.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:30.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:30.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:30.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:30.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:30.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:30.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:30.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:30.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:30.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:30.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:31.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:31.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:31.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:31.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:31.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:31.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:31.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:31.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:31.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:32.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598583 2024-09-16T16:22:32.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598583 '!=' null 2024-09-16T16:22:32.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:32.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:32.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:32.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:32.200 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:32.201 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:32.201 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:32.201 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:32.201 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:32.201 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:32.201 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:32.201 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:32.201 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:32.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:32.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:32.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:32.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:32.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:32.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:32.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:32.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:32.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:32.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:32.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:32.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:32.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:33.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:33.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:33.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:33.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:33.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:33.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:33.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:33.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:33.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:33.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598077 2024-09-16T16:22:33.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598077 '!=' null 2024-09-16T16:22:33.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:33.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:33.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:33.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:33.982 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:33.983 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:33.984 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:33.984 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:33.984 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:33.984 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:33.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:34.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:34.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:34.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:34.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:34.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:34.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:34.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:34.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:34.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:34.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:34.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:34.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:35.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:35.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:35.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:35.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:35.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:35.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:35.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:35.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:35.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:36.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333332 2024-09-16T16:22:36.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333332 '!=' null 2024-09-16T16:22:36.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:36.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:36.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:36.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:36.134 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:36.134 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:36.134 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:36.134 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:36.134 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:36.134 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:36.134 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:36.134 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:36.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:36.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:36.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:36.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:36.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:36.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:36.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:36.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:36.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:36.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:36.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:36.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:36.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:37.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:37.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:37.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:37.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:37.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:37.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:37.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:37.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:37.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:37.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471090 2024-09-16T16:22:37.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471090 '!=' null 2024-09-16T16:22:37.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:37.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:37.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:37.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:38.049 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:38.050 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:38.051 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:38.051 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:38.051 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:38.051 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:38.051 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:38.051 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:38.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:38.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:38.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:38.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:38.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:38.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:38.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:38.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:38.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:38.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:38.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:38.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:38.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:39.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:39.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:39.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:39.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:39.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:39.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:39.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:39.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:39.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:39.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333228 2024-09-16T16:22:39.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333228 '!=' null 2024-09-16T16:22:39.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:39.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:39.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:39.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:39.984 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:39.984 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:39.984 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:39.984 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:39.984 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:39.984 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:39.984 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:39.984 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:39.984 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:39.984 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:39.985 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:39.985 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:39.985 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:39.985 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:39.985 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:39.985 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:39.985 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:39.985 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:39.985 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:39.985 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:39.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:40.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:40.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:40.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:40.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:40.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:40.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:40.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:40.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:40.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:40.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:40.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:40.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:41.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:41.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:41.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:41.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:41.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:41.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:41.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:41.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:41.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:41.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597739 2024-09-16T16:22:41.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597739 '!=' null 2024-09-16T16:22:41.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:41.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:41.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:41.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:41.909 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:41.910 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:41.911 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:41.911 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:41.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:42.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:42.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:42.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:42.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:42.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:42.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:42.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:42.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:42.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:42.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:42.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:42.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:43.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:43.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:43.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:43.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:43.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:43.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:43.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:43.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:43.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:43.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597664 2024-09-16T16:22:43.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597664 '!=' null 2024-09-16T16:22:43.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:43.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:43.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:43.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:43.783 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:43.783 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:43.785 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:43.785 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:43.785 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:43.785 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:43.785 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:43.785 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:43.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:43.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:43.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:43.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:43.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:43.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:43.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:43.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:43.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:44.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:44.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:44.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:44.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:44.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:44.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:44.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:44.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:44.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:44.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:44.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:44.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:44.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:45.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460171 2024-09-16T16:22:45.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460171 '!=' null 2024-09-16T16:22:45.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:45.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:45.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:45.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:45.577 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:45.578 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:45.578 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:45.578 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:45.578 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:45.578 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:45.578 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:45.578 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:45.578 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:45.579 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:45.579 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:45.579 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:45.579 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:45.579 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:45.579 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:45.579 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:45.579 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:45.579 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:45.579 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:45.579 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:45.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:45.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:45.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:45.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:45.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:45.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:45.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:45.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:45.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:46.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:46.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:46.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:46.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:46.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:46.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:46.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:46.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:46.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:46.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:46.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:46.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:46.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:47.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724489 2024-09-16T16:22:47.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724489 '!=' null 2024-09-16T16:22:47.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:47.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:47.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:47.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:47.398 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:47.399 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:47.400 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:47.400 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:47.400 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:47.400 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:47.400 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:47.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:47.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:47.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:47.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:47.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:47.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:47.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:47.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:47.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:47.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:47.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:47.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:47.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:48.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:48.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:48.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:48.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:48.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:48.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:48.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:48.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:48.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:49.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724489 2024-09-16T16:22:49.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724489 '!=' null 2024-09-16T16:22:49.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:49.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:49.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:49.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:49.209 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:49.209 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:49.209 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:49.209 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:49.210 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:49.211 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:49.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:49.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:49.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:49.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:49.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:49.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:49.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:49.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:49.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:49.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:49.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:49.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:49.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:50.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:50.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:50.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:50.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:50.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:50.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:50.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:50.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:50.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:50.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460125 2024-09-16T16:22:50.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460125 '!=' null 2024-09-16T16:22:50.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:50.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:50.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:50.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:51.033 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:51.034 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:51.034 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:51.034 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:51.034 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:51.034 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:51.034 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:51.034 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:51.034 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:51.034 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:51.034 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:51.035 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:51.035 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:51.035 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:51.035 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:51.035 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:51.035 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:51.035 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:51.035 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:51.035 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:51.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:51.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:51.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:51.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:51.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:51.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:51.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:51.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:51.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:51.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:51.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:51.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:51.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:52.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:52.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:52.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:52.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:52.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:52.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:52.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:52.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:52.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597892 2024-09-16T16:22:52.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597892 '!=' null 2024-09-16T16:22:52.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:52.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:52.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:52.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:52.948 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:52.948 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:52.948 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:52.948 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:52.948 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:52.948 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:52.948 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:52.948 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:52.948 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:52.949 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:52.949 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:52.949 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:52.949 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:52.949 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:52.949 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:52.949 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:52.949 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:52.949 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:52.949 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:52.949 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:52.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:53.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:53.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:53.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:53.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:53.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:53.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:53.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:53.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:53.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:53.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:53.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:53.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:54.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:54.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:54.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:54.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:54.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:54.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:54.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:54.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:54.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:54.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598069 2024-09-16T16:22:54.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598069 '!=' null 2024-09-16T16:22:54.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:54.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:54.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:54.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:54.707 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:54.708 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:54.708 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:54.708 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:54.708 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:54.708 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:54.708 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:54.708 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:54.708 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:54.708 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:54.708 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:54.709 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:54.709 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:54.709 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:54.709 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:54.709 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:54.709 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:54.709 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:54.709 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:54.709 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:54.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:54.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:54.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:54.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:54.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:54.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:54.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:54.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:54.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:55.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:55.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:55.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:55.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:55.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:55.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:55.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:55.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:55.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:55.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:55.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:55.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:55.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:56.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460096 2024-09-16T16:22:56.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460096 '!=' null 2024-09-16T16:22:56.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:56.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:56.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:56.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:56.534 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:56.534 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:56.535 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:56.536 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:56.536 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:56.536 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:56.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:56.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:56.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:56.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:56.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:56.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:56.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:56.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:56.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:57.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:57.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:57.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:57.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:57.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:57.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:57.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:57.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:57.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:57.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:57.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:57.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:22:57.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:58.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598115 2024-09-16T16:22:58.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598115 '!=' null 2024-09-16T16:22:58.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:22:58.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:58.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:22:58.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:22:58.487 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:22:58.487 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:22:58.488 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:22:58.489 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:22:58.489 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:22:58.489 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:22:58.489 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:22:58.489 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:22:58.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:22:58.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:22:58.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:22:58.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:22:58.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:22:58.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:22:58.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:22:58.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:22:58.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:22:59.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:22:59.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:22:59.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:22:59.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:22:59.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:22:59.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:22:59.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:22:59.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:22:59.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:22:59.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:22:59.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:22:59.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:22:59.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:00.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333746 2024-09-16T16:23:00.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333746 '!=' null 2024-09-16T16:23:00.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:00.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:00.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:00.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:00.369 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:00.370 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:00.370 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:00.370 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:00.370 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:00.370 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:00.370 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:00.370 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:00.370 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:00.370 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:00.370 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:00.370 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:00.371 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:00.371 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:00.371 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:00.371 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:00.371 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:00.371 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:00.371 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:00.371 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:00.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:00.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:00.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:00.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:00.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:00.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:00.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:00.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:00.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:00.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:00.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:00.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:00.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:01.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:01.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:01.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:01.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:01.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:01.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:01.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:01.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:01.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:02.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597949 2024-09-16T16:23:02.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597949 '!=' null 2024-09-16T16:23:02.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:02.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:02.126 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:02.126 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:02.239 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:02.239 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:02.239 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:02.239 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:02.239 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:02.239 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:02.239 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:02.240 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:02.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:02.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:02.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:02.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:02.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:02.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:02.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:02.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:02.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:02.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:02.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:02.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:02.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:03.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:03.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:03.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:03.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:03.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:03.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:03.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:03.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:03.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:04.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598246 2024-09-16T16:23:04.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598246 '!=' null 2024-09-16T16:23:04.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:04.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:04.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:04.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:04.143 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:04.143 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:04.143 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:04.143 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:04.143 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:04.143 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:04.144 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:04.144 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:04.144 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:04.144 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:04.144 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:04.144 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:04.144 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:04.144 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:04.144 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:04.144 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:04.145 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:04.145 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:04.145 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:04.145 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:04.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:04.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:04.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:04.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:04.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:04.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:04.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:04.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:04.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:04.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:04.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:04.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:04.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:05.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:05.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:05.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:05.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:05.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:05.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:05.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:05.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:05.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:05.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333696 2024-09-16T16:23:05.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333696 '!=' null 2024-09-16T16:23:05.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:05.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:05.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:05.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:06.021 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:06.021 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:06.021 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:06.021 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:06.021 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:06.021 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:06.021 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:06.021 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:06.021 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:06.022 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:06.022 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:06.022 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:06.022 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:06.022 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:06.022 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:06.022 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:06.022 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:06.022 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:06.022 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:06.022 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:06.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:06.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:06.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:06.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:06.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:06.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:06.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:06.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:06.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:06.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:06.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:06.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:06.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:07.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:07.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:07.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:07.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:07.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:07.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:07.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:07.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:07.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:07.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471662 2024-09-16T16:23:07.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471662 '!=' null 2024-09-16T16:23:07.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:07.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:07.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:07.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:07.884 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:07.884 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:07.884 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:07.884 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:07.884 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:07.884 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:07.884 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:07.885 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:07.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:07.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:07.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:07.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:07.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:07.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:07.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:07.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:08.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:08.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:08.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:08.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:08.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:09.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:09.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:09.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:09.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:09.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:09.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:09.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:09.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:09.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:09.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333836 2024-09-16T16:23:09.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333836 '!=' null 2024-09-16T16:23:09.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:09.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:09.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:09.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:09.738 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:09.738 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:09.738 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:09.738 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:09.738 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:09.738 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:09.738 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:09.738 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:09.738 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:09.739 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:09.739 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:09.739 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:09.739 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:09.739 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:09.739 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:09.739 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:09.739 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:09.739 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:09.739 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:09.739 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:09.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:09.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:09.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:09.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:09.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:09.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:09.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:09.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:09.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:10.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:10.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:10.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:10.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:11.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:11.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:11.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:11.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:11.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:11.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:11.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:11.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:11.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:11.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598352 2024-09-16T16:23:11.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598352 '!=' null 2024-09-16T16:23:11.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:11.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:11.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:11.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:11.723 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:11.724 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:11.724 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:11.724 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:11.724 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:11.724 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:11.724 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:11.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:11.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:11.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:11.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:11.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:11.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:11.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:11.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:11.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:12.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:12.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:12.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:12.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:12.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:12.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:12.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:12.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:12.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:12.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:12.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:12.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:12.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:13.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598297 2024-09-16T16:23:13.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598297 '!=' null 2024-09-16T16:23:13.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:13.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:13.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:13.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:13.568 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:13.568 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:13.568 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:13.568 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:13.569 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:13.569 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:13.569 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:13.569 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:13.569 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:13.569 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:13.569 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:13.569 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:13.569 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:13.569 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:13.570 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:13.570 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:13.570 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:13.570 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:13.570 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:13.570 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:13.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:13.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:13.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:13.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:13.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:13.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:13.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:13.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:13.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:14.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:14.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:14.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:14.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:14.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:14.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:14.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:15.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460215 2024-09-16T16:23:15.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460215 '!=' null 2024-09-16T16:23:15.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:15.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:15.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:15.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:15.422 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:15.422 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:15.422 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:15.422 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:15.422 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:15.422 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:15.422 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:15.422 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:15.422 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:15.422 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:15.422 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:15.423 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:15.423 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:15.423 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:15.423 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:15.423 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:15.423 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:15.423 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:15.423 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:15.423 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:15.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:15.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:15.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:15.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:15.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:15.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:15.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:15.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:15.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:15.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:15.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:15.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:15.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:16.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:16.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:16.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:16.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:16.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:16.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:16.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:16.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:16.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460215 2024-09-16T16:23:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460215 '!=' null 2024-09-16T16:23:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:17.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:17.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:17.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:17.256 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:17.256 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:17.256 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:17.256 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:17.256 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:17.256 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:17.256 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:17.256 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:17.256 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:17.256 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:17.257 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:17.257 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:17.257 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:17.257 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:17.257 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:17.257 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:17.257 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:17.257 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:17.257 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:17.257 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:17.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:17.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:17.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:17.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:17.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:17.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:17.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:17.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:17.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:17.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:17.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:17.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:17.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:18.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:18.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:18.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:18.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:18.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:18.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:18.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:18.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:18.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:18.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724354 2024-09-16T16:23:18.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724354 '!=' null 2024-09-16T16:23:18.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:18.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:18.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:18.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:19.088 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:19.088 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:19.088 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:19.088 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:19.088 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:19.088 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:19.088 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:19.088 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:19.088 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:19.089 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:19.089 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:19.089 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:19.089 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:19.089 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:19.089 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:19.089 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:19.089 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:19.089 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:19.089 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:19.089 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:19.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:19.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:19.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:19.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:19.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:19.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:19.203 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:19.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:19.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:19.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:19.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:20.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:20.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:20.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:20.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:20.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:20.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:20.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:20.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:20.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:20.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460188 2024-09-16T16:23:20.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460188 '!=' null 2024-09-16T16:23:20.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:20.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:20.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:20.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:21.093 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:21.093 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:21.094 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:21.094 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:21.094 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:21.094 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:21.094 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:21.094 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:21.094 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:21.094 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:21.094 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:21.094 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:21.094 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:21.095 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:21.095 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:21.095 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:21.095 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:21.095 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:21.095 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:21.095 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:21.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:21.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:21.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:21.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:21.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:21.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:21.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:21.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:21.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:21.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:21.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:21.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:21.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:22.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:22.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:22.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:22.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:22.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:22.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:22.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:22.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:22.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:22.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598019 2024-09-16T16:23:22.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598019 '!=' null 2024-09-16T16:23:22.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:22.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:22.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:22.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:22.993 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:22.993 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:22.993 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:22.993 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:22.993 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:22.993 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:22.993 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:22.993 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:22.994 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:23.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:23.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:23.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:23.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:23.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:23.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:23.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:23.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:23.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:23.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:23.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:23.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:23.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:24.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:24.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:24.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:24.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:24.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:24.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:24.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:24.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:24.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:24.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597976 2024-09-16T16:23:24.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597976 '!=' null 2024-09-16T16:23:24.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:24.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:24.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:24.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:24.877 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:24.877 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:24.877 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:24.877 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:24.877 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:24.877 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:24.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:24.877 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:24.877 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:24.877 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:24.877 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:24.878 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:24.878 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:24.878 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:24.878 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:24.878 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:24.878 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:24.878 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:24.878 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:24.878 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:24.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:24.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:24.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:24.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:24.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:24.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:24.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:24.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:24.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:25.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:25.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:25.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:25.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:26.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:26.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:26.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:26.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:26.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:26.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:26.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:26.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:26.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:26.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460310 2024-09-16T16:23:26.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460310 '!=' null 2024-09-16T16:23:26.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:26.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:26.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:26.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:26.931 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:26.931 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:26.931 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:26.931 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:26.931 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:26.932 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:26.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:27.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:27.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:27.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:27.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:27.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:27.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:27.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:27.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:27.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:27.500 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:27.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:27.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:28.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:28.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:28.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:28.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:28.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:28.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:28.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:28.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:28.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:28.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597923 2024-09-16T16:23:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597923 '!=' null 2024-09-16T16:23:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:28.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:28.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:28.773 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:28.773 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:28.773 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:28.773 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:28.773 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:28.773 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:28.773 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:28.773 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:28.773 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:28.773 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:28.774 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:28.774 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:28.774 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:28.774 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:28.774 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:28.774 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:28.774 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:28.774 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:28.774 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:28.774 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:28.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:28.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:28.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:28.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:28.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:28.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:28.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:28.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:28.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:29.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:29.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:29.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:29.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:29.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:29.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:29.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:29.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:30.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460184 2024-09-16T16:23:30.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460184 '!=' null 2024-09-16T16:23:30.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:30.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:30.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:30.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:30.537 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:30.537 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:30.537 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:30.538 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:30.538 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:30.538 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:30.538 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:30.538 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:30.538 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:30.538 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:30.538 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:30.538 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:30.538 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:30.538 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:30.539 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:30.539 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:30.539 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:30.539 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:30.539 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:30.539 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:30.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:30.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:30.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:30.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:30.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:30.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:30.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:30.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:30.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:31.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:31.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:31.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:31.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:31.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:31.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:31.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:31.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:31.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:31.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:31.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:31.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:31.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:32.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724525 2024-09-16T16:23:32.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724525 '!=' null 2024-09-16T16:23:32.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:32.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:32.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:32.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:32.575 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:32.577 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:32.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:32.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:32.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:32.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:32.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:32.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:32.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:32.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:32.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:33.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:33.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:33.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:33.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:33.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:33.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:33.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:33.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:33.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:33.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:33.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:33.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:33.770 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:34.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724331 2024-09-16T16:23:34.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724331 '!=' null 2024-09-16T16:23:34.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:34.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:34.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:34.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:34.434 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:34.434 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:34.434 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:34.434 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:34.434 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:34.434 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:34.435 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:34.435 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:34.435 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:34.435 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:34.435 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:34.435 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:34.435 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:34.435 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:34.435 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:34.435 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:34.436 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:34.436 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:34.436 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:34.436 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:34.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:34.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:34.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:34.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:34.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:34.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:34.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:34.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:34.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:34.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:34.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:34.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:35.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:35.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:35.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:35.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:35.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:35.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:35.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:35.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:35.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:36.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586172 2024-09-16T16:23:36.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586172 '!=' null 2024-09-16T16:23:36.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:36.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:36.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:36.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:36.432 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:36.432 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:36.432 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:36.432 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:36.432 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:36.432 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:36.432 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:36.432 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:36.432 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:36.432 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:36.433 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:36.433 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:36.433 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:36.433 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:36.433 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:36.433 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:36.433 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:36.433 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:36.433 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:36.433 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:36.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:36.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:36.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:36.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:36.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:36.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:36.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:36.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:36.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:36.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:36.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:36.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:36.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:37.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:37.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:37.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:37.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:37.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:37.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:37.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:37.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:37.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:38.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724281 2024-09-16T16:23:38.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724281 '!=' null 2024-09-16T16:23:38.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:38.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:38.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:38.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:38.517 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:38.517 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:38.518 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:38.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:38.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:38.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:38.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:38.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:38.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:38.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:38.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:38.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:39.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:39.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:39.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:39.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:39.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:39.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:39.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:39.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:39.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:40.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460174 2024-09-16T16:23:40.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460174 '!=' null 2024-09-16T16:23:40.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:40.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:40.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:40.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:40.353 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:40.353 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:40.353 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:40.353 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:40.353 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:40.353 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:40.353 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:40.353 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:40.354 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:40.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:40.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:40.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:40.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:40.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:40.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:40.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:40.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:40.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:40.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:40.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:40.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:40.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:41.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:41.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:41.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:41.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:41.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:41.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:41.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:41.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:41.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:41.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598003 2024-09-16T16:23:41.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598003 '!=' null 2024-09-16T16:23:41.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:41.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:41.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:41.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:42.103 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:42.103 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:42.103 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:42.103 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:42.103 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:42.103 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:42.104 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:42.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:42.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:42.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:42.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:42.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:42.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:42.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:42.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:42.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:42.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:42.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:42.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:42.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:43.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:43.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:43.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:43.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598123 2024-09-16T16:23:43.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598123 '!=' null 2024-09-16T16:23:43.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:43.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:43.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:43.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:43.940 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:43.941 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:43.941 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:43.941 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:43.941 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:43.941 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:43.941 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:43.941 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:43.941 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:43.941 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:43.941 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:43.941 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:43.942 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:43.942 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:43.942 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:43.942 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:43.942 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:43.942 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:43.942 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:43.942 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:43.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:44.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:44.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:44.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:44.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:44.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:44.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:44.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:44.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:44.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:44.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:44.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:44.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:45.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:45.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:45.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:45.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:45.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:45.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:45.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:45.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:45.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:45.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460372 2024-09-16T16:23:45.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460372 '!=' null 2024-09-16T16:23:45.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:45.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:45.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:45.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:45.782 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:45.782 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:45.782 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:45.782 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:45.783 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:45.784 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:45.784 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:45.784 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:45.784 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:45.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:45.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:45.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:45.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:45.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:45.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:45.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:45.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:45.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:46.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:46.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:46.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:46.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:47.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:47.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:47.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:47.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:47.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:47.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:47.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:47.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:47.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:47.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598087 2024-09-16T16:23:47.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598087 '!=' null 2024-09-16T16:23:47.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:47.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:47.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:47.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:47.718 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:47.718 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:47.718 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:47.718 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:47.718 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:47.718 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:47.718 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:47.718 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:47.718 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:47.718 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:47.718 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:47.719 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:47.719 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:47.719 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:47.719 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:47.719 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:47.719 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:47.719 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:47.719 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:47.719 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:47.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:47.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:47.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:47.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:47.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:47.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:47.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:47.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:47.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:48.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:48.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:48.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:48.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:48.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:48.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:48.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:48.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:48.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:48.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:48.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:48.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:48.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:49.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333767 2024-09-16T16:23:49.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333767 '!=' null 2024-09-16T16:23:49.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:49.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:49.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:49.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:49.516 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:49.516 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:49.516 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:49.516 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:49.516 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:49.516 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:49.517 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:49.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:49.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:49.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:49.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:49.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:49.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:49.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:49.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:49.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:50.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:50.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:50.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:50.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:50.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:50.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:50.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:50.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:50.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:50.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:50.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:50.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:50.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:51.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598000 2024-09-16T16:23:51.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598000 '!=' null 2024-09-16T16:23:51.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:51.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:51.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:51.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:51.427 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:51.428 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:51.428 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:51.428 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:51.428 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:51.428 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:51.428 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:51.428 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:51.428 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:51.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:51.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:51.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:51.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:51.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:51.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:51.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:51.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:51.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:51.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:51.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:51.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:51.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:52.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:52.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:52.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:52.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:52.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:52.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:52.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:52.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:52.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:53.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598205 2024-09-16T16:23:53.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598205 '!=' null 2024-09-16T16:23:53.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:53.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:53.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:53.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:53.470 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:53.471 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:53.472 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:53.472 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:53.472 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:53.472 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:53.472 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:53.472 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:53.472 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:53.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:53.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:53.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:53.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:53.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:53.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:53.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:54.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:54.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:54.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:54.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:54.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:54.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:54.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:54.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:54.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:54.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:54.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:54.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:54.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:55.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598205 2024-09-16T16:23:55.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598205 '!=' null 2024-09-16T16:23:55.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:55.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:55.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:55.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:55.300 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:55.301 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:55.302 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:55.302 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:55.302 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:55.302 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:55.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:55.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:55.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:55.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:55.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:55.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:55.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:55.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:55.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:55.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:55.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:55.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:55.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:56.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:56.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:57.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460367 2024-09-16T16:23:57.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460367 '!=' null 2024-09-16T16:23:57.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:57.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:57.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:57.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:57.225 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:57.225 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:57.225 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:57.225 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:57.225 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:57.225 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:57.225 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:57.225 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:57.225 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:57.225 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:57.225 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:57.226 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:57.226 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:57.226 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:57.226 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:57.226 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:57.226 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:57.226 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:57.226 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:57.226 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:57.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:57.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:57.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:57.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:57.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:57.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:57.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:57.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:57.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:57.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:57.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:57.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:57.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:23:58.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:23:58.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:23:58.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:23:58.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:23:58.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:23:58.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:23:58.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:23:58.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:23:58.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:23:58.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597880 2024-09-16T16:23:58.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597880 '!=' null 2024-09-16T16:23:58.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:23:58.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:58.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:23:58.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:23:58.976 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:23:58.977 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:23:58.978 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:23:58.978 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:23:58.978 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:23:58.978 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:23:58.978 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:23:58.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:23:59.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:23:59.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:23:59.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:23:59.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:23:59.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:23:59.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:23:59.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:23:59.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:23:59.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:23:59.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:23:59.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:23:59.620 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:24:00.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:24:00.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:24:00.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:24:00.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:24:00.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:24:00.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:24:00.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:24:00.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:24:00.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:24:00.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333597 2024-09-16T16:24:00.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333597 '!=' null 2024-09-16T16:24:00.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:24:00.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:00.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:24:00.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:01.053 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:24:01.053 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:24:01.053 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:01.053 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:01.053 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:01.053 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:24:01.053 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:01.053 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:24:01.053 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:24:01.053 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:24:01.054 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:01.054 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:01.054 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:01.054 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:24:01.054 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:01.054 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:01.054 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:01.054 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:24:01.054 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:01.054 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:24:01.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:24:01.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:24:01.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:24:01.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:24:01.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:24:01.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:24:01.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:24:01.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:24:01.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:24:01.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:24:01.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:24:01.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:24:01.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:24:02.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:24:02.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:24:02.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:24:02.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:24:02.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:24:02.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:24:02.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:24:02.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:24:02.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:24:02.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597751 2024-09-16T16:24:02.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597751 '!=' null 2024-09-16T16:24:02.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:24:02.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:02.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:24:02.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:03.031 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:24:03.031 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [ 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: "item": "3.0", 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: "prio": 190, 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: "can_preempt": true 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:24:03.032 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:03.033 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:03.033 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:03.033 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:24:03.033 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:03.033 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:24:03.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:24:03.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:24:03.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:24:03.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:24:03.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:24:03.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:24:03.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:24:03.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:24:03.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:24:03.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:24:03.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:24:03.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:24:03.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:24:04.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:24:04.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:24:04.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:24:04.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:24:04.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:24:04.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:24:04.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:24:04.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:24:04.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:24:04.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597839 2024-09-16T16:24:04.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597839 '!=' null 2024-09-16T16:24:04.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:24:04.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:04.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:24:04.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:04.935 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:24:04.935 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:24:04.935 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:04.935 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:04.935 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:04.936 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:24:04.936 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:04.936 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:24:04.936 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:04.936 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:04.936 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:04.936 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:24:04.936 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:04.936 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:24:04.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:24:05.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:24:05.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:24:05.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:24:05.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:24:05.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:24:05.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:24:05.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:24:05.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:24:05.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:24:05.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:24:05.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:24:05.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:24:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:24:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:24:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:24:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:24:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:24:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:24:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:24:06.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:24:06.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:24:06.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24896249 2024-09-16T16:24:06.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24896249 '!=' null 2024-09-16T16:24:06.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:24:06.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:06.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:24:06.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:06.739 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:24:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:06.741 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:24:06.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:24:06.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:24:06.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:24:06.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:24:06.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:24:06.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:24:06.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:24:06.857 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:24:06.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:24:07.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:24:07.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:24:07.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:24:07.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:24:07.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:24:07.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:24:07.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:24:07.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:24:07.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:24:07.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:24:07.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:24:07.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:24:07.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:24:08.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11793309 2024-09-16T16:24:08.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11793309 '!=' null 2024-09-16T16:24:08.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:24:08.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:08.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:24:08.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:08.467 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:24:08.467 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:24:08.467 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:08.467 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:08.467 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:08.467 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:24:08.467 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:08.467 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:24:08.467 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:08.467 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:08.467 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:08.468 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:24:08.468 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:08.468 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:24:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:24:08.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:24:08.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:24:08.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:24:08.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:24:08.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:24:08.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:24:08.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:24:08.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:24:08.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T16:24:08.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:24:08.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:24:08.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:24:09.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-16T16:24:09.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-16T16:24:09.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:24:09.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:24:09.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:24:09.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:24:09.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:24:09.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:24:09.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:24:10.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11793722 2024-09-16T16:24:10.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11793722 '!=' null 2024-09-16T16:24:10.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:24:10.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:10.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-16T16:24:10.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_recovery_reservations 2024-09-16T16:24:10.185 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:24:10.186 INFO:tasks.workunit.client.0.smithi039.stdout: "local_reservations": { 2024-09-16T16:24:10.186 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:10.186 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:10.186 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:10.186 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:24:10.186 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:10.186 INFO:tasks.workunit.client.0.smithi039.stdout: "remote_reservations": { 2024-09-16T16:24:10.186 INFO:tasks.workunit.client.0.smithi039.stdout: "max_allowed": 1, 2024-09-16T16:24:10.186 INFO:tasks.workunit.client.0.smithi039.stdout: "min_priority": 0, 2024-09-16T16:24:10.186 INFO:tasks.workunit.client.0.smithi039.stdout: "queues": [], 2024-09-16T16:24:10.187 INFO:tasks.workunit.client.0.smithi039.stdout: "in_progress": [] 2024-09-16T16:24:10.187 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:10.187 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:24:10.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:24:10.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:24:10.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:24:10.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:24:10.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:24:10.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:24:10.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:24:10.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:24:10.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:24:10.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-16T16:24:10.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:24:10.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:24:10.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:24:11.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-09-16T16:24:11.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:24:11.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:24:11.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:298: TEST_recovery_priority: ceph pg dump pgs 2024-09-16T16:24:11.696 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T16:24:11.697 INFO:tasks.workunit.client.0.smithi039.stdout:7.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-16T16:20:31.310285+0000 96'600 122:1101 [1,4] 1 [1,4] 1 0'0 2024-09-16T16:12:15.456943+0000 0'0 2024-09-16T16:12:15.456943+0000 0 0 periodic scrub scheduled @ 2024-09-18T00:49:08.186329+0000 0 0 2024-09-16T16:24:11.697 INFO:tasks.workunit.client.0.smithi039.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-16T16:24:04.171223+0000 96'600 123:1117 [1,2] 1 [1,2] 1 0'0 2024-09-16T16:12:03.866975+0000 0'0 2024-09-16T16:12:03.866975+0000 0 0 periodic scrub scheduled @ 2024-09-17T16:24:19.821000+0000 0 0 2024-09-16T16:24:11.697 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-16T16:22:18.776806+0000 96'600 122:1106 [1,2] 1 [1,2] 1 0'0 2024-09-16T16:11:58.123419+0000 0'0 2024-09-16T16:11:58.123419+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:43:07.079516+0000 0 0 2024-09-16T16:24:11.697 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:24:11.697 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T16:24:11.697 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:24:11.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: get_asok_path osd.1 2024-09-16T16:24:11.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T16:24:11.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T16:24:11.713 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:24:11.713 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:24:11.713 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:24:11.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203397/ceph-osd.1.asok 2024-09-16T16:24:11.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: CEPH_ARGS= 2024-09-16T16:24:11.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.203397/ceph-osd.1.asok dump_pgstate_history 2024-09-16T16:24:11.815 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: "pgs": [ 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: "pg": "7.0", 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: "history": [ 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "65", 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:16.212492+0000", 2024-09-16T16:24:11.816 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:16.212512+0000" 2024-09-16T16:24:11.817 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.817 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.817 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.817 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:16.212525+0000", 2024-09-16T16:24:11.817 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:16.333304+0000" 2024-09-16T16:24:11.817 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.817 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.817 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.818 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:16.333305+0000", 2024-09-16T16:24:11.818 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:16.333522+0000" 2024-09-16T16:24:11.818 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.818 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.818 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-16T16:24:11.818 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:16.333523+0000", 2024-09-16T16:24:11.818 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:16.333569+0000" 2024-09-16T16:24:11.818 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.818 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.819 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-16T16:24:11.819 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:16.333570+0000", 2024-09-16T16:24:11.819 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:16.530846+0000" 2024-09-16T16:24:11.819 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.819 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.819 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.819 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:16.212519+0000", 2024-09-16T16:24:11.819 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:16.530873+0000" 2024-09-16T16:24:11.819 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.819 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.820 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Activating", 2024-09-16T16:24:11.820 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:16.531493+0000", 2024-09-16T16:24:11.820 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:17.084472+0000" 2024-09-16T16:24:11.820 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.820 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.820 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-16T16:24:11.820 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:17.084473+0000", 2024-09-16T16:24:11.820 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:17.084534+0000" 2024-09-16T16:24:11.820 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.821 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.821 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Clean", 2024-09-16T16:24:11.821 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:17.084534+0000", 2024-09-16T16:24:11.821 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.222557+0000" 2024-09-16T16:24:11.821 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.821 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.821 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active", 2024-09-16T16:24:11.821 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:16.530874+0000", 2024-09-16T16:24:11.821 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.222585+0000" 2024-09-16T16:24:11.822 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.822 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.822 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.822 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:16.212513+0000", 2024-09-16T16:24:11.822 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.222603+0000" 2024-09-16T16:24:11.822 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.822 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.822 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.822 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:16.212485+0000", 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.222606+0000" 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "66", 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.222606+0000", 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.222890+0000" 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.823 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "94", 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.222896+0000", 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.222906+0000" 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.222912+0000", 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.324306+0000" 2024-09-16T16:24:11.824 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.825 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.825 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.825 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.324307+0000", 2024-09-16T16:24:11.825 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.324419+0000" 2024-09-16T16:24:11.825 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.825 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.825 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-16T16:24:11.825 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.324420+0000", 2024-09-16T16:24:11.825 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.324441+0000" 2024-09-16T16:24:11.826 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.826 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.826 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-16T16:24:11.826 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.324442+0000", 2024-09-16T16:24:11.826 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.683888+0000" 2024-09-16T16:24:11.826 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.826 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.826 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.222909+0000", 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.683906+0000" 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Activating", 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.684180+0000", 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.761509+0000" 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.761510+0000", 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:18.761596+0000" 2024-09-16T16:24:11.827 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Clean", 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.761596+0000", 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.861522+0000" 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active", 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.683907+0000", 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.861559+0000" 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.222906+0000", 2024-09-16T16:24:11.828 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.861600+0000" 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:18.222890+0000", 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.861613+0000" 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "94", 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.829 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.861614+0000", 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.862197+0000" 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "101", 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.862206+0000", 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.862234+0000" 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.830 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.862252+0000", 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.862495+0000" 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.862496+0000", 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.862622+0000" 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.862623+0000", 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.862650+0000" 2024-09-16T16:24:11.831 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.862651+0000", 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:47.479731+0000" 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.862244+0000", 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:47.479738+0000" 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Activating", 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:47.479813+0000", 2024-09-16T16:24:11.832 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:47.538839+0000" 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:47.538840+0000", 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:47.538992+0000" 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Clean", 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:47.538993+0000", 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:52.718549+0000" 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.833 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active", 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:47.479739+0000", 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:52.718609+0000" 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.862235+0000", 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:52.718871+0000" 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.862197+0000", 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:52.718887+0000" 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "101", 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:52.718889+0000", 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:52.720342+0000" 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "103", 2024-09-16T16:24:11.835 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:52.720363+0000", 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:52.720422+0000" 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:52.720464+0000", 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:52.828238+0000" 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:52.828240+0000", 2024-09-16T16:24:11.836 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:52.828473+0000" 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:52.720441+0000", 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:52.828517+0000" 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:52.828518+0000", 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:53.582266+0000" 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:52.720423+0000", 2024-09-16T16:24:11.837 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:53.582324+0000" 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:52.720343+0000", 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:53.582334+0000" 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "103", 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:53.582335+0000", 2024-09-16T16:24:11.838 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:53.583023+0000" 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "118", 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:53.583034+0000", 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:53.583065+0000" 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.839 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:53.583084+0000", 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:53.584221+0000" 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:53.584223+0000", 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:53.584384+0000" 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:53.584385+0000", 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:53.584424+0000" 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.840 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:53.584424+0000", 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:53.942996+0000" 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:53.583075+0000", 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:53.943020+0000" 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Activating", 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:53.946057+0000", 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:54.134880+0000" 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.841 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:54.134882+0000", 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:54.135219+0000" 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:54.135220+0000", 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:17:54.136081+0000" 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:54.136082+0000", 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:29.869449+0000" 2024-09-16T16:24:11.842 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:29.869449+0000", 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:29.869669+0000" 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:29.869669+0000", 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:41.659434+0000" 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:41.659434+0000", 2024-09-16T16:24:11.843 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:41.660006+0000" 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:41.660006+0000", 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:20:29.477669+0000" 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:20:29.477670+0000", 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:20:29.903051+0000" 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.844 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active", 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:53.943021+0000", 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:20:29.903137+0000" 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:53.583066+0000", 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:20:29.903185+0000" 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:17:53.583024+0000", 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:20:29.903203+0000" 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "118", 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:20:29.903205+0000", 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:20:29.904099+0000" 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.846 INFO:tasks.workunit.client.0.smithi039.stdout: "pg": "3.0", 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: "history": [ 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "41", 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:04.211017+0000", 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:04.211035+0000" 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:04.211047+0000", 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:04.352992+0000" 2024-09-16T16:24:11.847 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:04.352994+0000", 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:04.353204+0000" 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:04.353207+0000", 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:04.353252+0000" 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:04.353252+0000", 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:04.679463+0000" 2024-09-16T16:24:11.848 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:04.211042+0000", 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:04.679478+0000" 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Activating", 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:04.679854+0000", 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:05.142303+0000" 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:05.142305+0000", 2024-09-16T16:24:11.849 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:05.142461+0000" 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Clean", 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:05.142462+0000", 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.222662+0000" 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active", 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:04.679480+0000", 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.222701+0000" 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.850 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:04.211036+0000", 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.222743+0000" 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:04.211011+0000", 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.222750+0000" 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "42", 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.851 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.222751+0000", 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.223396+0000" 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "91", 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.223399+0000", 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.223409+0000" 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.852 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.223415+0000", 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.332290+0000" 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.332291+0000", 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.332357+0000" 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.332358+0000", 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.332371+0000" 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-16T16:24:11.853 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.332371+0000", 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.624751+0000" 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.223412+0000", 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.624779+0000" 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Activating", 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.625044+0000", 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.886493+0000" 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-16T16:24:11.854 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.886494+0000", 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:06.886581+0000" 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Clean", 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.886582+0000", 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:45.874937+0000" 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active", 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.624780+0000", 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:45.874974+0000" 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.855 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.223409+0000", 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:45.875006+0000" 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:06.223396+0000", 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:45.875013+0000" 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "92", 2024-09-16T16:24:11.856 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:45.875014+0000", 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:45.875669+0000" 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "109", 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:45.875677+0000", 2024-09-16T16:24:11.857 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:45.875705+0000" 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:45.875721+0000", 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.003176+0000" 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.003178+0000", 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.003320+0000" 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-16T16:24:11.858 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.003321+0000", 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.003351+0000" 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.003352+0000", 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.245622+0000" 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:45.875713+0000", 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.245642+0000" 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Activating", 2024-09-16T16:24:11.859 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.245783+0000", 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.347604+0000" 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.347606+0000", 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:46.347748+0000" 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Clean", 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.347749+0000", 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.412993+0000" 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active", 2024-09-16T16:24:11.860 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:46.245643+0000", 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.413057+0000" 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:45.875706+0000", 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.413264+0000" 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:45.875670+0000", 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.413272+0000" 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.861 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "110", 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.413274+0000", 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.414374+0000" 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "112", 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.862 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.414387+0000", 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.414419+0000" 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.414437+0000", 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.521973+0000" 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.521974+0000", 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.522114+0000" 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.863 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.414428+0000", 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.522133+0000" 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.522134+0000", 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.974784+0000" 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.414420+0000", 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.974839+0000" 2024-09-16T16:24:11.864 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.414375+0000", 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.974850+0000" 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "112", 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.974851+0000", 2024-09-16T16:24:11.865 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.975548+0000" 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "122", 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.975561+0000", 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.975591+0000" 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.975613+0000", 2024-09-16T16:24:11.866 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.976771+0000" 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.976772+0000", 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.976933+0000" 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.976934+0000", 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:14.976973+0000" 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-16T16:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.976974+0000", 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:15.486485+0000" 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.975603+0000", 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:15.486506+0000" 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Activating", 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:15.489720+0000", 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:15.730333+0000" 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.868 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:15.730335+0000", 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:29.869602+0000" 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:29.869602+0000", 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:29.870287+0000" 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:29.870289+0000", 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:41.659255+0000" 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.869 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:41.659256+0000", 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:41.659496+0000" 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:41.659496+0000", 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:22:16.848815+0000" 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:22:16.848816+0000", 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:22:16.849207+0000" 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.870 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:22:16.849208+0000", 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:24:03.278686+0000" 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:24:03.278687+0000", 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:24:03.512133+0000" 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active", 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:15.486507+0000", 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:24:03.512173+0000" 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.871 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.975591+0000", 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:24:03.512224+0000" 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:14.975549+0000", 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:24:03.512239+0000" 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.872 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "122", 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:24:03.512240+0000", 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:24:03.513049+0000" 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: "pg": "1.0", 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: "history": [ 2024-09-16T16:24:11.873 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "27", 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Initial", 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:58.945930+0000", 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:11:58.946301+0000" 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "27", 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.874 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:58.946302+0000", 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:11:58.946375+0000" 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "29", 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:58.946382+0000", 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:11:58.946401+0000" 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.875 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:58.946415+0000", 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:11:59.063801+0000" 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:59.063803+0000", 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:11:59.063947+0000" 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:59.063950+0000", 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:11:59.063973+0000" 2024-09-16T16:24:11.876 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:59.063974+0000", 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:11:59.377142+0000" 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:58.946408+0000", 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:11:59.377159+0000" 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Activating", 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:59.377597+0000", 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:00.177392+0000" 2024-09-16T16:24:11.877 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active", 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:59.377161+0000", 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:00.177431+0000" 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:58.946402+0000", 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:00.177481+0000" 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:11:58.946376+0000", 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:00.177490+0000" 2024-09-16T16:24:11.878 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "29", 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:00.177491+0000", 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:00.178018+0000" 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.879 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "105", 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:00.178026+0000", 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:00.178045+0000" 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:00.178057+0000", 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:00.362383+0000" 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:00.362385+0000", 2024-09-16T16:24:11.880 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:00.362519+0000" 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:00.362520+0000", 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:00.362545+0000" 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:00.362546+0000", 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:00.843416+0000" 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.881 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:00.178051+0000", 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:00.843446+0000" 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Activating", 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:00.843908+0000", 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:01.511923+0000" 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:01.511925+0000", 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:12:01.512113+0000" 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.882 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Clean", 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:01.512114+0000", 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.362611+0000" 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active", 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:00.843447+0000", 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.362666+0000" 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:00.178046+0000", 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.362902+0000" 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.883 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:12:00.178019+0000", 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.362910+0000" 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "105", 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.362914+0000", 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.363960+0000" 2024-09-16T16:24:11.884 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "107", 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.363970+0000", 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.363998+0000" 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.885 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.886 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.364019+0000", 2024-09-16T16:24:11.886 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.480582+0000" 2024-09-16T16:24:11.886 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.886 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.886 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.886 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.480583+0000", 2024-09-16T16:24:11.886 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.480823+0000" 2024-09-16T16:24:11.886 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.886 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.886 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.886 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.364011+0000", 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.480862+0000" 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.480863+0000", 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.942252+0000" 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.363999+0000", 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.942301+0000" 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.887 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.363960+0000", 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.942311+0000" 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "107", 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.942312+0000", 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.943029+0000" 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.888 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "120", 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Start", 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.943042+0000", 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:06.943074+0000" 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.943092+0000", 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:07.078188+0000" 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.889 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:07.078190+0000", 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:07.078378+0000" 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:07.078379+0000", 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:07.078414+0000" 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:07.078415+0000", 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:07.428284+0000" 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.890 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Peering", 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.943083+0000", 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:07.428309+0000" 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Activating", 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:07.431874+0000", 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:18:07.949228+0000" 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:07.949230+0000", 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:20:29.477947+0000" 2024-09-16T16:24:11.891 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:20:29.477947+0000", 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:20:29.478645+0000" 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:20:29.478646+0000", 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:22:16.848542+0000" 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:22:16.848543+0000", 2024-09-16T16:24:11.892 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:22:17.400726+0000" 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary/Active", 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:07.428310+0000", 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:22:17.400782+0000" 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started/Primary", 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.943074+0000", 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:22:17.400833+0000" 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Started", 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:18:06.943029+0000", 2024-09-16T16:24:11.893 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:22:17.400844+0000" 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": "120", 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: "states": [ 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "Reset", 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: "enter": "2024-09-16T16:22:17.400845+0000", 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: "exit": "2024-09-16T16:22:17.401729+0000" 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.894 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T16:24:11.895 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T16:24:11.895 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T16:24:11.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:302: TEST_recovery_priority: '[' 0 '!=' 0 ']' 2024-09-16T16:24:11.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:306: TEST_recovery_priority: echo TEST PASSED 2024-09-16T16:24:11.895 INFO:tasks.workunit.client.0.smithi039.stdout:TEST PASSED 2024-09-16T16:24:11.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:309: TEST_recovery_priority: delete_pool test1 2024-09-16T16:24:11.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-09-16T16:24:11.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-09-16T16:24:12.403 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test1' does not exist 2024-09-16T16:24:12.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:310: TEST_recovery_priority: delete_pool test3 2024-09-16T16:24:12.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-09-16T16:24:12.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-09-16T16:24:13.084 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test3' does not exist 2024-09-16T16:24:13.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:311: TEST_recovery_priority: delete_pool test7 2024-09-16T16:24:13.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test7 2024-09-16T16:24:13.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2024-09-16T16:24:13.844 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test7' does not exist 2024-09-16T16:24:13.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:312: TEST_recovery_priority: kill_daemons td/osd-recovery-prio 2024-09-16T16:24:13.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:24:13.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:24:13.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:24:13.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:24:13.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:24:19.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:24:19.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:313: TEST_recovery_priority: return 0 2024-09-16T16:24:19.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-09-16T16:24:19.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-16T16:24:19.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:24:19.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-16T16:24:19.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:24:19.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:24:19.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:24:19.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:24:19.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:24:19.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:24:19.197 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:24:19.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:24:19.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:24:19.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:24:19.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:24:19.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:24:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:24:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:24:19.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:24:19.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:24:19.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:24:19.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:24:19.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-16T16:24:19.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:24:19.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:24:19.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:24:19.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.203397 2024-09-16T16:24:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:24:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:24:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T16:24:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-recovery-prio 0 2024-09-16T16:24:19.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-16T16:24:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T16:24:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-16T16:24:19.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:24:19.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:24:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:24:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:24:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:24:19.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:24:19.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:24:19.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:24:19.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:24:19.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:24:19.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:24:19.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:24:19.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:24:19.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:24:19.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:24:19.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:24:19.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:24:19.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T16:24:19.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-16T16:24:19.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:24:19.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:24:19.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203397 2024-09-16T16:24:19.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.203397 2024-09-16T16:24:19.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:24:19.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:24:19.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T16:24:19.296 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T16:24:19.296 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T16:24:19.348 INFO:tasks.workunit:Running workunit osd/osd-recovery-space.sh... 2024-09-16T16:24:19.349 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-recovery-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2024-09-16T16:24:19.399 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T16:24:19.404 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T16:24:19.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T16:24:19.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T16:24:19.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T16:24:19.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T16:24:19.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T16:24:19.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T16:24:19.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T16:24:19.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T16:24:19.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-recovery-space 2024-09-16T16:24:19.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:21: run: local dir=td/osd-recovery-space 2024-09-16T16:24:19.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:22: run: shift 2024-09-16T16:24:19.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: export CEPH_MON=127.0.0.1:7221 2024-09-16T16:24:19.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: CEPH_MON=127.0.0.1:7221 2024-09-16T16:24:19.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:25: run: export CEPH_ARGS 2024-09-16T16:24:19.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: uuidgen 2024-09-16T16:24:19.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: CEPH_ARGS+='--fsid=f5249a10-5ed5-4225-b027-aef69043c2f6 --auth-supported=none ' 2024-09-16T16:24:19.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7221 ' 2024-09-16T16:24:19.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:28: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2024-09-16T16:24:19.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-09-16T16:24:19.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: export objects=600 2024-09-16T16:24:19.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: objects=600 2024-09-16T16:24:19.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: export poolprefix=test 2024-09-16T16:24:19.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: poolprefix=test 2024-09-16T16:24:19.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: set 2024-09-16T16:24:19.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T16:24:19.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: local funcs=TEST_recovery_test_simple 2024-09-16T16:24:19.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:34: run: for func in $funcs 2024-09-16T16:24:19.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:35: run: setup td/osd-recovery-space 2024-09-16T16:24:19.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-space 2024-09-16T16:24:19.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-space 2024-09-16T16:24:19.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-09-16T16:24:19.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:24:19.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-09-16T16:24:19.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:24:19.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:24:19.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:24:19.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:24:19.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:24:19.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:24:19.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:24:19.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:24:19.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:24:19.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:24:19.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:24:19.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:24:19.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:24:19.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:24:19.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:24:19.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:24:19.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:24:19.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:24:19.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-09-16T16:24:19.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:24:19.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:24:19.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258350 2024-09-16T16:24:19.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.258350 2024-09-16T16:24:19.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:24:19.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:24:19.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-space 2024-09-16T16:24:19.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:24:19.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:24:19.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258350 2024-09-16T16:24:19.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.258350 2024-09-16T16:24:19.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:24:19.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T16:24:19.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T16:24:19.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:24:19.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-space 1' TERM HUP INT 2024-09-16T16:24:19.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:36: run: TEST_recovery_test_simple td/osd-recovery-space 2024-09-16T16:24:19.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:89: TEST_recovery_test_simple: local dir=td/osd-recovery-space 2024-09-16T16:24:19.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:90: TEST_recovery_test_simple: local pools=1 2024-09-16T16:24:19.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:91: TEST_recovery_test_simple: local OSDS=2 2024-09-16T16:24:19.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:93: TEST_recovery_test_simple: run_mon td/osd-recovery-space a 2024-09-16T16:24:19.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-space 2024-09-16T16:24:19.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:24:19.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:24:19.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:24:19.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-space/a 2024-09-16T16:24:19.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-space/a --run-dir=td/osd-recovery-space 2024-09-16T16:24:19.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:24:19.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:24:19.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:24:19.775 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:24:19.775 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:24:19.775 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258350 2024-09-16T16:24:19.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258350/$cluster-$name.asok' 2024-09-16T16:24:19.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-space/a '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.258350/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-space/log --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:24:19.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:24:19.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:24:19.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:24:19.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:24:19.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:24:19.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:24:19.997 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:24:19.997 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:24:19.997 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:24:19.997 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:24:19.997 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:24:19.997 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258350 2024-09-16T16:24:19.997 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.258350/ceph-mon.a.asok 2024-09-16T16:24:19.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:24:19.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.258350/ceph-mon.a.asok config get fsid 2024-09-16T16:24:20.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:24:20.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:24:20.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:24:20.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:24:20.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:24:20.117 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:24:20.117 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:24:20.117 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:24:20.117 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:24:20.117 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:24:20.117 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258350 2024-09-16T16:24:20.117 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.258350/ceph-mon.a.asok 2024-09-16T16:24:20.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:24:20.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.258350/ceph-mon.a.asok config get mon_host 2024-09-16T16:24:20.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:94: TEST_recovery_test_simple: run_mgr td/osd-recovery-space x 2024-09-16T16:24:20.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-space 2024-09-16T16:24:20.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:24:20.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:24:20.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:24:20.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-space/x 2024-09-16T16:24:20.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:24:21.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:24:21.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:24:21.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:24:21.186 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:24:21.186 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:24:21.186 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258350 2024-09-16T16:24:21.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258350/$cluster-$name.asok' 2024-09-16T16:24:21.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:24:21.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-space/x '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.258350/$cluster-$name.asok' --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:24:21.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:95: TEST_recovery_test_simple: export CEPH_ARGS 2024-09-16T16:24:21.241 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: expr 2 - 1 2024-09-16T16:24:21.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: seq 0 1 2024-09-16T16:24:21.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T16:24:21.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 0 2024-09-16T16:24:21.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-09-16T16:24:21.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:24:21.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:24:21.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:24:21.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/0 2024-09-16T16:24:21.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f5249a10-5ed5-4225-b027-aef69043c2f6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-09-16T16:24:21.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:24:21.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/0' 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/0/journal' 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258350 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258350/$cluster-$name.asok' 2024-09-16T16:24:21.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.258350/$cluster-$name.asok' 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:24:21.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/0 2024-09-16T16:24:21.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:24:21.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0cc9afc1-7737-4306-abc0-740c71de37e5 2024-09-16T16:24:21.248 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 0cc9afc1-7737-4306-abc0-740c71de37e5 2024-09-16T16:24:21.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0cc9afc1-7737-4306-abc0-740c71de37e5' 2024-09-16T16:24:21.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:24:21.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1W+hmGmTsDxAAprtHhpyzcBw8nd+Aq68/mA== 2024-09-16T16:24:21.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1W+hmGmTsDxAAprtHhpyzcBw8nd+Aq68/mA=="}' 2024-09-16T16:24:21.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0cc9afc1-7737-4306-abc0-740c71de37e5 -i td/osd-recovery-space/0/new.json 2024-09-16T16:24:21.606 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:24:21.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/0/new.json 2024-09-16T16:24:21.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f5249a10-5ed5-4225-b027-aef69043c2f6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.258350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1W+hmGmTsDxAAprtHhpyzcBw8nd+Aq68/mA== --osd-uuid 0cc9afc1-7737-4306-abc0-740c71de37e5 2024-09-16T16:24:21.653 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:21.651+0000 7ff32193c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:24:21.657 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:21.659+0000 7ff32193c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:24:21.658 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:21.659+0000 7ff32193c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:24:21.659 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:21.659+0000 7ff32193c7c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-09-16T16:24:21.659 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:21.659+0000 7ff32193c7c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-09-16T16:24:21.659 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:21.659+0000 7ff32193c7c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-09-16T16:24:21.697 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:21.699+0000 7ff32193c7c0 -1 bluestore(td/osd-recovery-space/0) _read_fsid unparsable uuid 2024-09-16T16:24:27.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/0/keyring 2024-09-16T16:24:27.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:24:27.853 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:24:27.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:24:27.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:24:28.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:24:28.424 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:24:28.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f5249a10-5ed5-4225-b027-aef69043c2f6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.258350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:24:28.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:24:28.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:24:28.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:24:28.451 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:28.451+0000 7fe0dcc4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:24:28.462 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:28.463+0000 7fe0dcc4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:24:28.464 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:28.463+0000 7fe0dcc4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:24:28.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:24:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:24:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:24:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:24:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:24:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:28.847 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:24:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:24:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:29.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:29.039+0000 7fe0dcc4b7c0 -1 Falling back to public interface 2024-09-16T16:24:29.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:30.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:30.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:30.268 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:24:30.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:24:30.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:30.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:30.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:31.579 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:31.578+0000 7fe0dcc4b7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:24:31.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:31.703 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:24:31.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:31.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:24:31.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:31.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:32.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:33.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:33.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:33.242 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:24:33.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:24:33.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:33.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:33.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:34.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:34.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:34.670 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:24:34.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:24:34.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:34.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:35.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:36.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:36.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:36.116 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:24:36.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:24:36.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:36.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:36.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:37.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:37.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:37.552 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:24:37.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:24:37.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:37.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:38.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:39.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:39.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:39.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:24:39.040 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:24:39.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:39.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:39.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:40.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:40.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:40.563 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:24:40.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:24:40.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:40.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:41.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:42.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:42.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:42.042 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:24:42.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:24:42.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:42.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:42.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:43.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:43.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:43.605 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:24:43.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:24:43.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:43.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:44.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:45.279 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:45.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:46.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:46.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:46.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:24:46.905 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:24:46.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:46.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:47.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:48.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:48.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:48.710 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:24:48.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:24:48.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:48.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:49.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:50.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:50.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:50.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:24:50.680 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:24:50.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:50.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:51.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:52.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:52.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:52.443 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:24:52.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:24:52.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:52.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:53.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:24:54.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:24:54.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:24:54.562 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:24:54.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:24:54.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:24:54.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:24:55.009 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2148064985,v1:127.0.0.1:6803/2148064985] [v2:127.0.0.1:6804/2148064985,v1:127.0.0.1:6805/2148064985] exists,up 0cc9afc1-7737-4306-abc0-740c71de37e5 2024-09-16T16:24:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:24:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:24:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:24:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T16:24:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 1 2024-09-16T16:24:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-09-16T16:24:55.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/1 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f5249a10-5ed5-4225-b027-aef69043c2f6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/1' 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/1/journal' 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-09-16T16:24:55.013 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258350 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258350/$cluster-$name.asok' 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.258350/$cluster-$name.asok' 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:24:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:24:55.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:24:55.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:24:55.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:24:55.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/1 2024-09-16T16:24:55.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:24:55.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0f513525-787e-4fef-b80b-4094bda72efc 2024-09-16T16:24:55.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0f513525-787e-4fef-b80b-4094bda72efc' 2024-09-16T16:24:55.016 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 0f513525-787e-4fef-b80b-4094bda72efc 2024-09-16T16:24:55.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:24:55.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXW+hmM59TAhAAUOyg9DJWpudhzShb3dTnhw== 2024-09-16T16:24:55.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXW+hmM59TAhAAUOyg9DJWpudhzShb3dTnhw=="}' 2024-09-16T16:24:55.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0f513525-787e-4fef-b80b-4094bda72efc -i td/osd-recovery-space/1/new.json 2024-09-16T16:24:55.733 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:24:55.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/1/new.json 2024-09-16T16:24:55.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f5249a10-5ed5-4225-b027-aef69043c2f6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.258350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXW+hmM59TAhAAUOyg9DJWpudhzShb3dTnhw== --osd-uuid 0f513525-787e-4fef-b80b-4094bda72efc 2024-09-16T16:24:55.791 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:55.790+0000 7fbe10aee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:24:55.793 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:55.794+0000 7fbe10aee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:24:55.795 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:55.794+0000 7fbe10aee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:24:55.795 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:55.794+0000 7fbe10aee7c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-09-16T16:24:55.795 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:55.794+0000 7fbe10aee7c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-09-16T16:24:55.796 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:55.794+0000 7fbe10aee7c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-09-16T16:24:55.864 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:24:55.862+0000 7fbe10aee7c0 -1 bluestore(td/osd-recovery-space/1) _read_fsid unparsable uuid 2024-09-16T16:25:01.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/1/keyring 2024-09-16T16:25:01.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:25:01.932 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:25:01.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:25:01.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:25:02.511 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:25:02.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:25:02.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:25:02.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f5249a10-5ed5-4225-b027-aef69043c2f6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.258350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:25:02.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:25:02.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:25:02.546 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:25:02.546+0000 7fb5a6f6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:25:02.572 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:25:02.570+0000 7fb5a6f6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:25:02.573 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:25:02.570+0000 7fb5a6f6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:25:02.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:25:02.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:25:02.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:25:02.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:25:02.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:25:02.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:02.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:25:02.912 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:25:02.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:02.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:03.139 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:25:03.138+0000 7fb5a6f6d7c0 -1 Falling back to public interface 2024-09-16T16:25:03.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:04.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:04.397 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:25:04.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:04.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:25:04.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:04.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:04.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:05.718 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:25:05.718+0000 7fb5a6f6d7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:25:05.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:05.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:05.827 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:25:05.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:25:05.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:05.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:06.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:07.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:07.345 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:25:07.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:07.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:25:07.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:07.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:07.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:08.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:08.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:08.807 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:25:08.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:25:08.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:08.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:09.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:10.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:10.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:10.245 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:25:10.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:25:10.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:10.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:10.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:11.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:11.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:11.924 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:25:11.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:25:11.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:11.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:12.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:13.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:13.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:13.544 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:25:13.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:25:13.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:13.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:13.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:15.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:15.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:15.002 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:25:15.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:25:15.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:15.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:15.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:16.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:16.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:16.534 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:25:16.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:25:16.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:16.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:17.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:18.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:18.103 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:25:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:25:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:18.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:19.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:20.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:20.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:20.163 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:25:20.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:25:20.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:20.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:21.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:22.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:22.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:22.053 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:25:22.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:25:22.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:22.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:22.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:23.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:23.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:23.508 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:25:23.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:25:23.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:23.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:24.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:25.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:25.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:25.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:25:25.275 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:25:25.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:25.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:26.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:27.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:27.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:27.071 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:25:27.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:25:27.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:27.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:27.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:25:28.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:25:28.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:25:28.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:25:28.787 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:25:28.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:25:28.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:25:29.299 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1268776955,v1:127.0.0.1:6811/1268776955] [v2:127.0.0.1:6812/1268776955,v1:127.0.0.1:6813/1268776955] exists,up 0f513525-787e-4fef-b80b-4094bda72efc 2024-09-16T16:25:29.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:25:29.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:25:29.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:25:29.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:102: TEST_recovery_test_simple: ceph osd set-nearfull-ratio .40 2024-09-16T16:25:30.199 INFO:tasks.workunit.client.0.smithi039.stderr:osd set-nearfull-ratio 0.4 2024-09-16T16:25:30.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: ceph osd set-backfillfull-ratio .45 2024-09-16T16:25:31.069 INFO:tasks.workunit.client.0.smithi039.stderr:osd set-backfillfull-ratio 0.45 2024-09-16T16:25:31.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:104: TEST_recovery_test_simple: ceph osd set-full-ratio .50 2024-09-16T16:25:31.781 INFO:tasks.workunit.client.0.smithi039.stderr:osd set-full-ratio 0.5 2024-09-16T16:25:31.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: seq 1 1 2024-09-16T16:25:31.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-09-16T16:25:31.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:108: TEST_recovery_test_simple: create_pool test1 1 1 2024-09-16T16:25:31.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-09-16T16:25:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test1' already exists 2024-09-16T16:25:32.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:25:33.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:109: TEST_recovery_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-09-16T16:25:34.200 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 1 2024-09-16T16:25:34.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:112: TEST_recovery_test_simple: wait_for_clean 2024-09-16T16:25:34.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:25:34.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:25:34.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:25:34.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:25:34.224 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:25:34.224 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:25:34.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:25:34.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:25:34.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:25:34.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:25:34.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:25:34.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:25:34.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:25:34.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:25:34.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:25:34.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:25:34.888 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-09-16T16:25:34.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:25:34.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:25:34.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:25:35.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836497 2024-09-16T16:25:35.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836497 2024-09-16T16:25:35.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836497' 2024-09-16T16:25:35.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:25:35.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:25:35.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672971 2024-09-16T16:25:35.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672971 2024-09-16T16:25:35.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836497 1-42949672971' 2024-09-16T16:25:35.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:25:35.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836497 2024-09-16T16:25:35.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:25:35.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:25:35.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836497 2024-09-16T16:25:35.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:25:35.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836497 2024-09-16T16:25:35.216 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836497 2024-09-16T16:25:35.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-09-16T16:25:35.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:25:35.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836494 -lt 21474836497 2024-09-16T16:25:35.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:25:36.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:25:36.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:25:37.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836497 -lt 21474836497 2024-09-16T16:25:37.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:25:37.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672971 2024-09-16T16:25:37.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:25:37.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:25:37.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672971 2024-09-16T16:25:37.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:25:37.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672971 2024-09-16T16:25:37.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-16T16:25:37.064 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672971 2024-09-16T16:25:37.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:25:37.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-16T16:25:37.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:25:37.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:25:37.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:25:38.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:25:38.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:25:38.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:25:38.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:25:38.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:25:38.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:25:38.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:25:38.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:25:38.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:25:38.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:25:38.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:25:38.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:25:38.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:25:38.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:25:38.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:25:38.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: dd if=/dev/urandom of=td/osd-recovery-space/datafile bs=1024 count=5 2024-09-16T16:25:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:5+0 records in 2024-09-16T16:25:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:5+0 records out 2024-09-16T16:25:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:5120 bytes (5.1 kB, 5.0 KiB) copied, 0.000135062 s, 37.9 MB/s 2024-09-16T16:25:38.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: seq 1 600 2024-09-16T16:25:38.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:38.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj1 td/osd-recovery-space/datafile 2024-09-16T16:25:39.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:39.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj2 td/osd-recovery-space/datafile 2024-09-16T16:25:39.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:39.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj3 td/osd-recovery-space/datafile 2024-09-16T16:25:39.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:39.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj4 td/osd-recovery-space/datafile 2024-09-16T16:25:39.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:39.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj5 td/osd-recovery-space/datafile 2024-09-16T16:25:39.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:39.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj6 td/osd-recovery-space/datafile 2024-09-16T16:25:39.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:39.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj7 td/osd-recovery-space/datafile 2024-09-16T16:25:39.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:39.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj8 td/osd-recovery-space/datafile 2024-09-16T16:25:39.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:39.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj9 td/osd-recovery-space/datafile 2024-09-16T16:25:39.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:39.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj10 td/osd-recovery-space/datafile 2024-09-16T16:25:39.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:39.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj11 td/osd-recovery-space/datafile 2024-09-16T16:25:40.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:40.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj12 td/osd-recovery-space/datafile 2024-09-16T16:25:40.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:40.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj13 td/osd-recovery-space/datafile 2024-09-16T16:25:40.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:40.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj14 td/osd-recovery-space/datafile 2024-09-16T16:25:40.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:40.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj15 td/osd-recovery-space/datafile 2024-09-16T16:25:40.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:40.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj16 td/osd-recovery-space/datafile 2024-09-16T16:25:40.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:40.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj17 td/osd-recovery-space/datafile 2024-09-16T16:25:40.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:40.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj18 td/osd-recovery-space/datafile 2024-09-16T16:25:40.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:40.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj19 td/osd-recovery-space/datafile 2024-09-16T16:25:40.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:40.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj20 td/osd-recovery-space/datafile 2024-09-16T16:25:41.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:41.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj21 td/osd-recovery-space/datafile 2024-09-16T16:25:41.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:41.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj22 td/osd-recovery-space/datafile 2024-09-16T16:25:41.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:41.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj23 td/osd-recovery-space/datafile 2024-09-16T16:25:41.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:41.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj24 td/osd-recovery-space/datafile 2024-09-16T16:25:41.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:41.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj25 td/osd-recovery-space/datafile 2024-09-16T16:25:41.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:41.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj26 td/osd-recovery-space/datafile 2024-09-16T16:25:41.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:41.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj27 td/osd-recovery-space/datafile 2024-09-16T16:25:41.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:41.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj28 td/osd-recovery-space/datafile 2024-09-16T16:25:41.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:41.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj29 td/osd-recovery-space/datafile 2024-09-16T16:25:41.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:41.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj30 td/osd-recovery-space/datafile 2024-09-16T16:25:41.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:41.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj31 td/osd-recovery-space/datafile 2024-09-16T16:25:42.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:42.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj32 td/osd-recovery-space/datafile 2024-09-16T16:25:42.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:42.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj33 td/osd-recovery-space/datafile 2024-09-16T16:25:42.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:42.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj34 td/osd-recovery-space/datafile 2024-09-16T16:25:42.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:42.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj35 td/osd-recovery-space/datafile 2024-09-16T16:25:42.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:42.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj36 td/osd-recovery-space/datafile 2024-09-16T16:25:42.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:42.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj37 td/osd-recovery-space/datafile 2024-09-16T16:25:42.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:42.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj38 td/osd-recovery-space/datafile 2024-09-16T16:25:42.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:42.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj39 td/osd-recovery-space/datafile 2024-09-16T16:25:42.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:42.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj40 td/osd-recovery-space/datafile 2024-09-16T16:25:43.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:43.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj41 td/osd-recovery-space/datafile 2024-09-16T16:25:43.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:43.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj42 td/osd-recovery-space/datafile 2024-09-16T16:25:43.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:43.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj43 td/osd-recovery-space/datafile 2024-09-16T16:25:43.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:43.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj44 td/osd-recovery-space/datafile 2024-09-16T16:25:43.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:43.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj45 td/osd-recovery-space/datafile 2024-09-16T16:25:43.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:43.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj46 td/osd-recovery-space/datafile 2024-09-16T16:25:43.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:43.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj47 td/osd-recovery-space/datafile 2024-09-16T16:25:43.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:43.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj48 td/osd-recovery-space/datafile 2024-09-16T16:25:43.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:43.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj49 td/osd-recovery-space/datafile 2024-09-16T16:25:43.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:43.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj50 td/osd-recovery-space/datafile 2024-09-16T16:25:43.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:43.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj51 td/osd-recovery-space/datafile 2024-09-16T16:25:44.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:44.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj52 td/osd-recovery-space/datafile 2024-09-16T16:25:44.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:44.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj53 td/osd-recovery-space/datafile 2024-09-16T16:25:44.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:44.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj54 td/osd-recovery-space/datafile 2024-09-16T16:25:44.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:44.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj55 td/osd-recovery-space/datafile 2024-09-16T16:25:44.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:44.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj56 td/osd-recovery-space/datafile 2024-09-16T16:25:44.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:44.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj57 td/osd-recovery-space/datafile 2024-09-16T16:25:44.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:44.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj58 td/osd-recovery-space/datafile 2024-09-16T16:25:44.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:44.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj59 td/osd-recovery-space/datafile 2024-09-16T16:25:44.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:44.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj60 td/osd-recovery-space/datafile 2024-09-16T16:25:45.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:45.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj61 td/osd-recovery-space/datafile 2024-09-16T16:25:45.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:45.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj62 td/osd-recovery-space/datafile 2024-09-16T16:25:45.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:45.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj63 td/osd-recovery-space/datafile 2024-09-16T16:25:45.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:45.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj64 td/osd-recovery-space/datafile 2024-09-16T16:25:45.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:45.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj65 td/osd-recovery-space/datafile 2024-09-16T16:25:45.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:45.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj66 td/osd-recovery-space/datafile 2024-09-16T16:25:45.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:45.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj67 td/osd-recovery-space/datafile 2024-09-16T16:25:45.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:45.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj68 td/osd-recovery-space/datafile 2024-09-16T16:25:45.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:45.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj69 td/osd-recovery-space/datafile 2024-09-16T16:25:45.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:45.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj70 td/osd-recovery-space/datafile 2024-09-16T16:25:45.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:45.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj71 td/osd-recovery-space/datafile 2024-09-16T16:25:46.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:46.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj72 td/osd-recovery-space/datafile 2024-09-16T16:25:46.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:46.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj73 td/osd-recovery-space/datafile 2024-09-16T16:25:46.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:46.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj74 td/osd-recovery-space/datafile 2024-09-16T16:25:46.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:46.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj75 td/osd-recovery-space/datafile 2024-09-16T16:25:46.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:46.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj76 td/osd-recovery-space/datafile 2024-09-16T16:25:46.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:46.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj77 td/osd-recovery-space/datafile 2024-09-16T16:25:46.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:46.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj78 td/osd-recovery-space/datafile 2024-09-16T16:25:46.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:46.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj79 td/osd-recovery-space/datafile 2024-09-16T16:25:46.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:46.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj80 td/osd-recovery-space/datafile 2024-09-16T16:25:47.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:47.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj81 td/osd-recovery-space/datafile 2024-09-16T16:25:47.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:47.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj82 td/osd-recovery-space/datafile 2024-09-16T16:25:47.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:47.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj83 td/osd-recovery-space/datafile 2024-09-16T16:25:47.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:47.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj84 td/osd-recovery-space/datafile 2024-09-16T16:25:47.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:47.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj85 td/osd-recovery-space/datafile 2024-09-16T16:25:47.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:47.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj86 td/osd-recovery-space/datafile 2024-09-16T16:25:47.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:47.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj87 td/osd-recovery-space/datafile 2024-09-16T16:25:47.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:47.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj88 td/osd-recovery-space/datafile 2024-09-16T16:25:47.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:47.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj89 td/osd-recovery-space/datafile 2024-09-16T16:25:47.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:47.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj90 td/osd-recovery-space/datafile 2024-09-16T16:25:47.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:47.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj91 td/osd-recovery-space/datafile 2024-09-16T16:25:48.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:48.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj92 td/osd-recovery-space/datafile 2024-09-16T16:25:48.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:48.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj93 td/osd-recovery-space/datafile 2024-09-16T16:25:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:48.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj94 td/osd-recovery-space/datafile 2024-09-16T16:25:48.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:48.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj95 td/osd-recovery-space/datafile 2024-09-16T16:25:48.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:48.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj96 td/osd-recovery-space/datafile 2024-09-16T16:25:48.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:48.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj97 td/osd-recovery-space/datafile 2024-09-16T16:25:48.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:48.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj98 td/osd-recovery-space/datafile 2024-09-16T16:25:48.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:48.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj99 td/osd-recovery-space/datafile 2024-09-16T16:25:48.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:48.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj100 td/osd-recovery-space/datafile 2024-09-16T16:25:49.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj101 td/osd-recovery-space/datafile 2024-09-16T16:25:49.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj102 td/osd-recovery-space/datafile 2024-09-16T16:25:49.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj103 td/osd-recovery-space/datafile 2024-09-16T16:25:49.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj104 td/osd-recovery-space/datafile 2024-09-16T16:25:49.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj105 td/osd-recovery-space/datafile 2024-09-16T16:25:49.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj106 td/osd-recovery-space/datafile 2024-09-16T16:25:49.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj107 td/osd-recovery-space/datafile 2024-09-16T16:25:49.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj108 td/osd-recovery-space/datafile 2024-09-16T16:25:49.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj109 td/osd-recovery-space/datafile 2024-09-16T16:25:49.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj110 td/osd-recovery-space/datafile 2024-09-16T16:25:49.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj111 td/osd-recovery-space/datafile 2024-09-16T16:25:49.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:49.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj112 td/osd-recovery-space/datafile 2024-09-16T16:25:50.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:50.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj113 td/osd-recovery-space/datafile 2024-09-16T16:25:50.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:50.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj114 td/osd-recovery-space/datafile 2024-09-16T16:25:50.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:50.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj115 td/osd-recovery-space/datafile 2024-09-16T16:25:50.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:50.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj116 td/osd-recovery-space/datafile 2024-09-16T16:25:50.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:50.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj117 td/osd-recovery-space/datafile 2024-09-16T16:25:50.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:50.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj118 td/osd-recovery-space/datafile 2024-09-16T16:25:50.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:50.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj119 td/osd-recovery-space/datafile 2024-09-16T16:25:50.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:50.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj120 td/osd-recovery-space/datafile 2024-09-16T16:25:50.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:50.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj121 td/osd-recovery-space/datafile 2024-09-16T16:25:51.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:51.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj122 td/osd-recovery-space/datafile 2024-09-16T16:25:51.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:51.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj123 td/osd-recovery-space/datafile 2024-09-16T16:25:51.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:51.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj124 td/osd-recovery-space/datafile 2024-09-16T16:25:51.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:51.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj125 td/osd-recovery-space/datafile 2024-09-16T16:25:51.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:51.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj126 td/osd-recovery-space/datafile 2024-09-16T16:25:51.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:51.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj127 td/osd-recovery-space/datafile 2024-09-16T16:25:51.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:51.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj128 td/osd-recovery-space/datafile 2024-09-16T16:25:51.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:51.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj129 td/osd-recovery-space/datafile 2024-09-16T16:25:51.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:51.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj130 td/osd-recovery-space/datafile 2024-09-16T16:25:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:51.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj131 td/osd-recovery-space/datafile 2024-09-16T16:25:51.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:51.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj132 td/osd-recovery-space/datafile 2024-09-16T16:25:52.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:52.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj133 td/osd-recovery-space/datafile 2024-09-16T16:25:52.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:52.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj134 td/osd-recovery-space/datafile 2024-09-16T16:25:52.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:52.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj135 td/osd-recovery-space/datafile 2024-09-16T16:25:52.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:52.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj136 td/osd-recovery-space/datafile 2024-09-16T16:25:52.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:52.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj137 td/osd-recovery-space/datafile 2024-09-16T16:25:52.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:52.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj138 td/osd-recovery-space/datafile 2024-09-16T16:25:52.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:52.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj139 td/osd-recovery-space/datafile 2024-09-16T16:25:52.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:52.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj140 td/osd-recovery-space/datafile 2024-09-16T16:25:52.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:52.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj141 td/osd-recovery-space/datafile 2024-09-16T16:25:53.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:53.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj142 td/osd-recovery-space/datafile 2024-09-16T16:25:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj143 td/osd-recovery-space/datafile 2024-09-16T16:25:53.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:53.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj144 td/osd-recovery-space/datafile 2024-09-16T16:25:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj145 td/osd-recovery-space/datafile 2024-09-16T16:25:53.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:53.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj146 td/osd-recovery-space/datafile 2024-09-16T16:25:53.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:53.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj147 td/osd-recovery-space/datafile 2024-09-16T16:25:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj148 td/osd-recovery-space/datafile 2024-09-16T16:25:53.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:53.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj149 td/osd-recovery-space/datafile 2024-09-16T16:25:53.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:53.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj150 td/osd-recovery-space/datafile 2024-09-16T16:25:53.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:53.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj151 td/osd-recovery-space/datafile 2024-09-16T16:25:53.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:53.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj152 td/osd-recovery-space/datafile 2024-09-16T16:25:54.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:54.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj153 td/osd-recovery-space/datafile 2024-09-16T16:25:54.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:54.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj154 td/osd-recovery-space/datafile 2024-09-16T16:25:54.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:54.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj155 td/osd-recovery-space/datafile 2024-09-16T16:25:54.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:54.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj156 td/osd-recovery-space/datafile 2024-09-16T16:25:54.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:54.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj157 td/osd-recovery-space/datafile 2024-09-16T16:25:54.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:54.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj158 td/osd-recovery-space/datafile 2024-09-16T16:25:54.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:54.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj159 td/osd-recovery-space/datafile 2024-09-16T16:25:54.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:54.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj160 td/osd-recovery-space/datafile 2024-09-16T16:25:54.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:54.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj161 td/osd-recovery-space/datafile 2024-09-16T16:25:54.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:54.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj162 td/osd-recovery-space/datafile 2024-09-16T16:25:55.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:55.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj163 td/osd-recovery-space/datafile 2024-09-16T16:25:55.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:55.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj164 td/osd-recovery-space/datafile 2024-09-16T16:25:55.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:55.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj165 td/osd-recovery-space/datafile 2024-09-16T16:25:55.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:55.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj166 td/osd-recovery-space/datafile 2024-09-16T16:25:55.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:55.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj167 td/osd-recovery-space/datafile 2024-09-16T16:25:55.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:55.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj168 td/osd-recovery-space/datafile 2024-09-16T16:25:55.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:55.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj169 td/osd-recovery-space/datafile 2024-09-16T16:25:55.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:55.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj170 td/osd-recovery-space/datafile 2024-09-16T16:25:56.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:56.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj171 td/osd-recovery-space/datafile 2024-09-16T16:25:56.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:56.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj172 td/osd-recovery-space/datafile 2024-09-16T16:25:56.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:56.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj173 td/osd-recovery-space/datafile 2024-09-16T16:25:56.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:56.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj174 td/osd-recovery-space/datafile 2024-09-16T16:25:56.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:56.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj175 td/osd-recovery-space/datafile 2024-09-16T16:25:56.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:56.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj176 td/osd-recovery-space/datafile 2024-09-16T16:25:56.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:56.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj177 td/osd-recovery-space/datafile 2024-09-16T16:25:56.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:56.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj178 td/osd-recovery-space/datafile 2024-09-16T16:25:57.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:57.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj179 td/osd-recovery-space/datafile 2024-09-16T16:25:57.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:57.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj180 td/osd-recovery-space/datafile 2024-09-16T16:25:57.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:57.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj181 td/osd-recovery-space/datafile 2024-09-16T16:25:57.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:57.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj182 td/osd-recovery-space/datafile 2024-09-16T16:25:57.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:57.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj183 td/osd-recovery-space/datafile 2024-09-16T16:25:57.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:57.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj184 td/osd-recovery-space/datafile 2024-09-16T16:25:57.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:57.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj185 td/osd-recovery-space/datafile 2024-09-16T16:25:57.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:57.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj186 td/osd-recovery-space/datafile 2024-09-16T16:25:57.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:57.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj187 td/osd-recovery-space/datafile 2024-09-16T16:25:57.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:57.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj188 td/osd-recovery-space/datafile 2024-09-16T16:25:57.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:57.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj189 td/osd-recovery-space/datafile 2024-09-16T16:25:58.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:58.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj190 td/osd-recovery-space/datafile 2024-09-16T16:25:58.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:58.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj191 td/osd-recovery-space/datafile 2024-09-16T16:25:58.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:58.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj192 td/osd-recovery-space/datafile 2024-09-16T16:25:58.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:58.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj193 td/osd-recovery-space/datafile 2024-09-16T16:25:58.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:58.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj194 td/osd-recovery-space/datafile 2024-09-16T16:25:58.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:58.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj195 td/osd-recovery-space/datafile 2024-09-16T16:25:58.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:58.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj196 td/osd-recovery-space/datafile 2024-09-16T16:25:58.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:58.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj197 td/osd-recovery-space/datafile 2024-09-16T16:25:58.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:58.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj198 td/osd-recovery-space/datafile 2024-09-16T16:25:58.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:58.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj199 td/osd-recovery-space/datafile 2024-09-16T16:25:59.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:59.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj200 td/osd-recovery-space/datafile 2024-09-16T16:25:59.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:59.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj201 td/osd-recovery-space/datafile 2024-09-16T16:25:59.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:59.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj202 td/osd-recovery-space/datafile 2024-09-16T16:25:59.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:59.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj203 td/osd-recovery-space/datafile 2024-09-16T16:25:59.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:59.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj204 td/osd-recovery-space/datafile 2024-09-16T16:25:59.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:59.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj205 td/osd-recovery-space/datafile 2024-09-16T16:25:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj206 td/osd-recovery-space/datafile 2024-09-16T16:25:59.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:59.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj207 td/osd-recovery-space/datafile 2024-09-16T16:25:59.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:59.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj208 td/osd-recovery-space/datafile 2024-09-16T16:25:59.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:25:59.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj209 td/osd-recovery-space/datafile 2024-09-16T16:26:00.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:00.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj210 td/osd-recovery-space/datafile 2024-09-16T16:26:00.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:00.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj211 td/osd-recovery-space/datafile 2024-09-16T16:26:00.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:00.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj212 td/osd-recovery-space/datafile 2024-09-16T16:26:00.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:00.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj213 td/osd-recovery-space/datafile 2024-09-16T16:26:00.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:00.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj214 td/osd-recovery-space/datafile 2024-09-16T16:26:00.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:00.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj215 td/osd-recovery-space/datafile 2024-09-16T16:26:00.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:00.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj216 td/osd-recovery-space/datafile 2024-09-16T16:26:00.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:00.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj217 td/osd-recovery-space/datafile 2024-09-16T16:26:00.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:00.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj218 td/osd-recovery-space/datafile 2024-09-16T16:26:00.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:00.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj219 td/osd-recovery-space/datafile 2024-09-16T16:26:01.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:01.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj220 td/osd-recovery-space/datafile 2024-09-16T16:26:01.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:01.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj221 td/osd-recovery-space/datafile 2024-09-16T16:26:01.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:01.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj222 td/osd-recovery-space/datafile 2024-09-16T16:26:01.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:01.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj223 td/osd-recovery-space/datafile 2024-09-16T16:26:01.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:01.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj224 td/osd-recovery-space/datafile 2024-09-16T16:26:01.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:01.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj225 td/osd-recovery-space/datafile 2024-09-16T16:26:01.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:01.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj226 td/osd-recovery-space/datafile 2024-09-16T16:26:01.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:01.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj227 td/osd-recovery-space/datafile 2024-09-16T16:26:01.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:01.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj228 td/osd-recovery-space/datafile 2024-09-16T16:26:01.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:01.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj229 td/osd-recovery-space/datafile 2024-09-16T16:26:02.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:02.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj230 td/osd-recovery-space/datafile 2024-09-16T16:26:02.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:02.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj231 td/osd-recovery-space/datafile 2024-09-16T16:26:02.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:02.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj232 td/osd-recovery-space/datafile 2024-09-16T16:26:02.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:02.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj233 td/osd-recovery-space/datafile 2024-09-16T16:26:02.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:02.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj234 td/osd-recovery-space/datafile 2024-09-16T16:26:02.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:02.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj235 td/osd-recovery-space/datafile 2024-09-16T16:26:02.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:02.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj236 td/osd-recovery-space/datafile 2024-09-16T16:26:02.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:02.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj237 td/osd-recovery-space/datafile 2024-09-16T16:26:02.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:02.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj238 td/osd-recovery-space/datafile 2024-09-16T16:26:02.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:02.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj239 td/osd-recovery-space/datafile 2024-09-16T16:26:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:03.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj240 td/osd-recovery-space/datafile 2024-09-16T16:26:03.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:03.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj241 td/osd-recovery-space/datafile 2024-09-16T16:26:03.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:03.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj242 td/osd-recovery-space/datafile 2024-09-16T16:26:03.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:03.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj243 td/osd-recovery-space/datafile 2024-09-16T16:26:03.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:03.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj244 td/osd-recovery-space/datafile 2024-09-16T16:26:03.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:03.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj245 td/osd-recovery-space/datafile 2024-09-16T16:26:03.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:03.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj246 td/osd-recovery-space/datafile 2024-09-16T16:26:03.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:03.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj247 td/osd-recovery-space/datafile 2024-09-16T16:26:03.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:03.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj248 td/osd-recovery-space/datafile 2024-09-16T16:26:03.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:03.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj249 td/osd-recovery-space/datafile 2024-09-16T16:26:03.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:03.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj250 td/osd-recovery-space/datafile 2024-09-16T16:26:04.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:04.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj251 td/osd-recovery-space/datafile 2024-09-16T16:26:04.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:04.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj252 td/osd-recovery-space/datafile 2024-09-16T16:26:04.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:04.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj253 td/osd-recovery-space/datafile 2024-09-16T16:26:04.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:04.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj254 td/osd-recovery-space/datafile 2024-09-16T16:26:04.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:04.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj255 td/osd-recovery-space/datafile 2024-09-16T16:26:04.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:04.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj256 td/osd-recovery-space/datafile 2024-09-16T16:26:04.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:04.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj257 td/osd-recovery-space/datafile 2024-09-16T16:26:04.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:04.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj258 td/osd-recovery-space/datafile 2024-09-16T16:26:04.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:04.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj259 td/osd-recovery-space/datafile 2024-09-16T16:26:04.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:04.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj260 td/osd-recovery-space/datafile 2024-09-16T16:26:05.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:05.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj261 td/osd-recovery-space/datafile 2024-09-16T16:26:05.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:05.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj262 td/osd-recovery-space/datafile 2024-09-16T16:26:05.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:05.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj263 td/osd-recovery-space/datafile 2024-09-16T16:26:05.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:05.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj264 td/osd-recovery-space/datafile 2024-09-16T16:26:05.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:05.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj265 td/osd-recovery-space/datafile 2024-09-16T16:26:05.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:05.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj266 td/osd-recovery-space/datafile 2024-09-16T16:26:05.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:05.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj267 td/osd-recovery-space/datafile 2024-09-16T16:26:05.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:05.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj268 td/osd-recovery-space/datafile 2024-09-16T16:26:05.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:05.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj269 td/osd-recovery-space/datafile 2024-09-16T16:26:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj270 td/osd-recovery-space/datafile 2024-09-16T16:26:06.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:06.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj271 td/osd-recovery-space/datafile 2024-09-16T16:26:06.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:06.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj272 td/osd-recovery-space/datafile 2024-09-16T16:26:06.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:06.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj273 td/osd-recovery-space/datafile 2024-09-16T16:26:06.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:06.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj274 td/osd-recovery-space/datafile 2024-09-16T16:26:06.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:06.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj275 td/osd-recovery-space/datafile 2024-09-16T16:26:06.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:06.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj276 td/osd-recovery-space/datafile 2024-09-16T16:26:06.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:06.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj277 td/osd-recovery-space/datafile 2024-09-16T16:26:06.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:06.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj278 td/osd-recovery-space/datafile 2024-09-16T16:26:06.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:06.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj279 td/osd-recovery-space/datafile 2024-09-16T16:26:06.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:06.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj280 td/osd-recovery-space/datafile 2024-09-16T16:26:07.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:07.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj281 td/osd-recovery-space/datafile 2024-09-16T16:26:07.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:07.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj282 td/osd-recovery-space/datafile 2024-09-16T16:26:07.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:07.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj283 td/osd-recovery-space/datafile 2024-09-16T16:26:07.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:07.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj284 td/osd-recovery-space/datafile 2024-09-16T16:26:07.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:07.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj285 td/osd-recovery-space/datafile 2024-09-16T16:26:07.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:07.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj286 td/osd-recovery-space/datafile 2024-09-16T16:26:07.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:07.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj287 td/osd-recovery-space/datafile 2024-09-16T16:26:07.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:07.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj288 td/osd-recovery-space/datafile 2024-09-16T16:26:07.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:07.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj289 td/osd-recovery-space/datafile 2024-09-16T16:26:07.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:07.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj290 td/osd-recovery-space/datafile 2024-09-16T16:26:08.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:08.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj291 td/osd-recovery-space/datafile 2024-09-16T16:26:08.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:08.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj292 td/osd-recovery-space/datafile 2024-09-16T16:26:08.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:08.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj293 td/osd-recovery-space/datafile 2024-09-16T16:26:08.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:08.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj294 td/osd-recovery-space/datafile 2024-09-16T16:26:08.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:08.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj295 td/osd-recovery-space/datafile 2024-09-16T16:26:08.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:08.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj296 td/osd-recovery-space/datafile 2024-09-16T16:26:08.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:08.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj297 td/osd-recovery-space/datafile 2024-09-16T16:26:08.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:08.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj298 td/osd-recovery-space/datafile 2024-09-16T16:26:08.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:08.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj299 td/osd-recovery-space/datafile 2024-09-16T16:26:09.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:09.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj300 td/osd-recovery-space/datafile 2024-09-16T16:26:09.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:09.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj301 td/osd-recovery-space/datafile 2024-09-16T16:26:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj302 td/osd-recovery-space/datafile 2024-09-16T16:26:09.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:09.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj303 td/osd-recovery-space/datafile 2024-09-16T16:26:09.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:09.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj304 td/osd-recovery-space/datafile 2024-09-16T16:26:09.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:09.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj305 td/osd-recovery-space/datafile 2024-09-16T16:26:09.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:09.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj306 td/osd-recovery-space/datafile 2024-09-16T16:26:09.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:09.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj307 td/osd-recovery-space/datafile 2024-09-16T16:26:09.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:09.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj308 td/osd-recovery-space/datafile 2024-09-16T16:26:09.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:09.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj309 td/osd-recovery-space/datafile 2024-09-16T16:26:09.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:09.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj310 td/osd-recovery-space/datafile 2024-09-16T16:26:10.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:10.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj311 td/osd-recovery-space/datafile 2024-09-16T16:26:10.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:10.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj312 td/osd-recovery-space/datafile 2024-09-16T16:26:10.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:10.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj313 td/osd-recovery-space/datafile 2024-09-16T16:26:10.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:10.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj314 td/osd-recovery-space/datafile 2024-09-16T16:26:10.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:10.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj315 td/osd-recovery-space/datafile 2024-09-16T16:26:10.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:10.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj316 td/osd-recovery-space/datafile 2024-09-16T16:26:10.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:10.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj317 td/osd-recovery-space/datafile 2024-09-16T16:26:10.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:10.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj318 td/osd-recovery-space/datafile 2024-09-16T16:26:10.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:10.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj319 td/osd-recovery-space/datafile 2024-09-16T16:26:11.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:11.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj320 td/osd-recovery-space/datafile 2024-09-16T16:26:11.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:11.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj321 td/osd-recovery-space/datafile 2024-09-16T16:26:11.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:11.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj322 td/osd-recovery-space/datafile 2024-09-16T16:26:11.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:11.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj323 td/osd-recovery-space/datafile 2024-09-16T16:26:11.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:11.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj324 td/osd-recovery-space/datafile 2024-09-16T16:26:11.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:11.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj325 td/osd-recovery-space/datafile 2024-09-16T16:26:11.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:11.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj326 td/osd-recovery-space/datafile 2024-09-16T16:26:11.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:11.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj327 td/osd-recovery-space/datafile 2024-09-16T16:26:11.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:11.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj328 td/osd-recovery-space/datafile 2024-09-16T16:26:11.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:11.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj329 td/osd-recovery-space/datafile 2024-09-16T16:26:11.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:11.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj330 td/osd-recovery-space/datafile 2024-09-16T16:26:12.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj331 td/osd-recovery-space/datafile 2024-09-16T16:26:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj332 td/osd-recovery-space/datafile 2024-09-16T16:26:12.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj333 td/osd-recovery-space/datafile 2024-09-16T16:26:12.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj334 td/osd-recovery-space/datafile 2024-09-16T16:26:12.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj335 td/osd-recovery-space/datafile 2024-09-16T16:26:12.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj336 td/osd-recovery-space/datafile 2024-09-16T16:26:12.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj337 td/osd-recovery-space/datafile 2024-09-16T16:26:12.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj338 td/osd-recovery-space/datafile 2024-09-16T16:26:12.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj339 td/osd-recovery-space/datafile 2024-09-16T16:26:12.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj340 td/osd-recovery-space/datafile 2024-09-16T16:26:12.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj341 td/osd-recovery-space/datafile 2024-09-16T16:26:12.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:12.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj342 td/osd-recovery-space/datafile 2024-09-16T16:26:13.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj343 td/osd-recovery-space/datafile 2024-09-16T16:26:13.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj344 td/osd-recovery-space/datafile 2024-09-16T16:26:13.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj345 td/osd-recovery-space/datafile 2024-09-16T16:26:13.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj346 td/osd-recovery-space/datafile 2024-09-16T16:26:13.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj347 td/osd-recovery-space/datafile 2024-09-16T16:26:13.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj348 td/osd-recovery-space/datafile 2024-09-16T16:26:13.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj349 td/osd-recovery-space/datafile 2024-09-16T16:26:13.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj350 td/osd-recovery-space/datafile 2024-09-16T16:26:13.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj351 td/osd-recovery-space/datafile 2024-09-16T16:26:13.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj352 td/osd-recovery-space/datafile 2024-09-16T16:26:13.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj353 td/osd-recovery-space/datafile 2024-09-16T16:26:13.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj354 td/osd-recovery-space/datafile 2024-09-16T16:26:13.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj355 td/osd-recovery-space/datafile 2024-09-16T16:26:13.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:13.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj356 td/osd-recovery-space/datafile 2024-09-16T16:26:14.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj357 td/osd-recovery-space/datafile 2024-09-16T16:26:14.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj358 td/osd-recovery-space/datafile 2024-09-16T16:26:14.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj359 td/osd-recovery-space/datafile 2024-09-16T16:26:14.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj360 td/osd-recovery-space/datafile 2024-09-16T16:26:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj361 td/osd-recovery-space/datafile 2024-09-16T16:26:14.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj362 td/osd-recovery-space/datafile 2024-09-16T16:26:14.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj363 td/osd-recovery-space/datafile 2024-09-16T16:26:14.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj364 td/osd-recovery-space/datafile 2024-09-16T16:26:14.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj365 td/osd-recovery-space/datafile 2024-09-16T16:26:14.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj366 td/osd-recovery-space/datafile 2024-09-16T16:26:14.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj367 td/osd-recovery-space/datafile 2024-09-16T16:26:14.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj368 td/osd-recovery-space/datafile 2024-09-16T16:26:14.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:14.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj369 td/osd-recovery-space/datafile 2024-09-16T16:26:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj370 td/osd-recovery-space/datafile 2024-09-16T16:26:15.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj371 td/osd-recovery-space/datafile 2024-09-16T16:26:15.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj372 td/osd-recovery-space/datafile 2024-09-16T16:26:15.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj373 td/osd-recovery-space/datafile 2024-09-16T16:26:15.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj374 td/osd-recovery-space/datafile 2024-09-16T16:26:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj375 td/osd-recovery-space/datafile 2024-09-16T16:26:15.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj376 td/osd-recovery-space/datafile 2024-09-16T16:26:15.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj377 td/osd-recovery-space/datafile 2024-09-16T16:26:15.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj378 td/osd-recovery-space/datafile 2024-09-16T16:26:15.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj379 td/osd-recovery-space/datafile 2024-09-16T16:26:15.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj380 td/osd-recovery-space/datafile 2024-09-16T16:26:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj381 td/osd-recovery-space/datafile 2024-09-16T16:26:15.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj382 td/osd-recovery-space/datafile 2024-09-16T16:26:15.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:15.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj383 td/osd-recovery-space/datafile 2024-09-16T16:26:16.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj384 td/osd-recovery-space/datafile 2024-09-16T16:26:16.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj385 td/osd-recovery-space/datafile 2024-09-16T16:26:16.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj386 td/osd-recovery-space/datafile 2024-09-16T16:26:16.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj387 td/osd-recovery-space/datafile 2024-09-16T16:26:16.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj388 td/osd-recovery-space/datafile 2024-09-16T16:26:16.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj389 td/osd-recovery-space/datafile 2024-09-16T16:26:16.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj390 td/osd-recovery-space/datafile 2024-09-16T16:26:16.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj391 td/osd-recovery-space/datafile 2024-09-16T16:26:16.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj392 td/osd-recovery-space/datafile 2024-09-16T16:26:16.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj393 td/osd-recovery-space/datafile 2024-09-16T16:26:16.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj394 td/osd-recovery-space/datafile 2024-09-16T16:26:16.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:16.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj395 td/osd-recovery-space/datafile 2024-09-16T16:26:17.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj396 td/osd-recovery-space/datafile 2024-09-16T16:26:17.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj397 td/osd-recovery-space/datafile 2024-09-16T16:26:17.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj398 td/osd-recovery-space/datafile 2024-09-16T16:26:17.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj399 td/osd-recovery-space/datafile 2024-09-16T16:26:17.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj400 td/osd-recovery-space/datafile 2024-09-16T16:26:17.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj401 td/osd-recovery-space/datafile 2024-09-16T16:26:17.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj402 td/osd-recovery-space/datafile 2024-09-16T16:26:17.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj403 td/osd-recovery-space/datafile 2024-09-16T16:26:17.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj404 td/osd-recovery-space/datafile 2024-09-16T16:26:17.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj405 td/osd-recovery-space/datafile 2024-09-16T16:26:17.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj406 td/osd-recovery-space/datafile 2024-09-16T16:26:17.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj407 td/osd-recovery-space/datafile 2024-09-16T16:26:17.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj408 td/osd-recovery-space/datafile 2024-09-16T16:26:17.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:17.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj409 td/osd-recovery-space/datafile 2024-09-16T16:26:18.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj410 td/osd-recovery-space/datafile 2024-09-16T16:26:18.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj411 td/osd-recovery-space/datafile 2024-09-16T16:26:18.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj412 td/osd-recovery-space/datafile 2024-09-16T16:26:18.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj413 td/osd-recovery-space/datafile 2024-09-16T16:26:18.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj414 td/osd-recovery-space/datafile 2024-09-16T16:26:18.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj415 td/osd-recovery-space/datafile 2024-09-16T16:26:18.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj416 td/osd-recovery-space/datafile 2024-09-16T16:26:18.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj417 td/osd-recovery-space/datafile 2024-09-16T16:26:18.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj418 td/osd-recovery-space/datafile 2024-09-16T16:26:18.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj419 td/osd-recovery-space/datafile 2024-09-16T16:26:18.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj420 td/osd-recovery-space/datafile 2024-09-16T16:26:18.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj421 td/osd-recovery-space/datafile 2024-09-16T16:26:18.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:18.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj422 td/osd-recovery-space/datafile 2024-09-16T16:26:19.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj423 td/osd-recovery-space/datafile 2024-09-16T16:26:19.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj424 td/osd-recovery-space/datafile 2024-09-16T16:26:19.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj425 td/osd-recovery-space/datafile 2024-09-16T16:26:19.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj426 td/osd-recovery-space/datafile 2024-09-16T16:26:19.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj427 td/osd-recovery-space/datafile 2024-09-16T16:26:19.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj428 td/osd-recovery-space/datafile 2024-09-16T16:26:19.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj429 td/osd-recovery-space/datafile 2024-09-16T16:26:19.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj430 td/osd-recovery-space/datafile 2024-09-16T16:26:19.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj431 td/osd-recovery-space/datafile 2024-09-16T16:26:19.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj432 td/osd-recovery-space/datafile 2024-09-16T16:26:19.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj433 td/osd-recovery-space/datafile 2024-09-16T16:26:19.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj434 td/osd-recovery-space/datafile 2024-09-16T16:26:19.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:19.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj435 td/osd-recovery-space/datafile 2024-09-16T16:26:20.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj436 td/osd-recovery-space/datafile 2024-09-16T16:26:20.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj437 td/osd-recovery-space/datafile 2024-09-16T16:26:20.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj438 td/osd-recovery-space/datafile 2024-09-16T16:26:20.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj439 td/osd-recovery-space/datafile 2024-09-16T16:26:20.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj440 td/osd-recovery-space/datafile 2024-09-16T16:26:20.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj441 td/osd-recovery-space/datafile 2024-09-16T16:26:20.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj442 td/osd-recovery-space/datafile 2024-09-16T16:26:20.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj443 td/osd-recovery-space/datafile 2024-09-16T16:26:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj444 td/osd-recovery-space/datafile 2024-09-16T16:26:20.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj445 td/osd-recovery-space/datafile 2024-09-16T16:26:20.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj446 td/osd-recovery-space/datafile 2024-09-16T16:26:20.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:20.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj447 td/osd-recovery-space/datafile 2024-09-16T16:26:21.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj448 td/osd-recovery-space/datafile 2024-09-16T16:26:21.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj449 td/osd-recovery-space/datafile 2024-09-16T16:26:21.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj450 td/osd-recovery-space/datafile 2024-09-16T16:26:21.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj451 td/osd-recovery-space/datafile 2024-09-16T16:26:21.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj452 td/osd-recovery-space/datafile 2024-09-16T16:26:21.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj453 td/osd-recovery-space/datafile 2024-09-16T16:26:21.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj454 td/osd-recovery-space/datafile 2024-09-16T16:26:21.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj455 td/osd-recovery-space/datafile 2024-09-16T16:26:21.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj456 td/osd-recovery-space/datafile 2024-09-16T16:26:21.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj457 td/osd-recovery-space/datafile 2024-09-16T16:26:21.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj458 td/osd-recovery-space/datafile 2024-09-16T16:26:21.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj459 td/osd-recovery-space/datafile 2024-09-16T16:26:21.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj460 td/osd-recovery-space/datafile 2024-09-16T16:26:21.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:21.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj461 td/osd-recovery-space/datafile 2024-09-16T16:26:22.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj462 td/osd-recovery-space/datafile 2024-09-16T16:26:22.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj463 td/osd-recovery-space/datafile 2024-09-16T16:26:22.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj464 td/osd-recovery-space/datafile 2024-09-16T16:26:22.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj465 td/osd-recovery-space/datafile 2024-09-16T16:26:22.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj466 td/osd-recovery-space/datafile 2024-09-16T16:26:22.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj467 td/osd-recovery-space/datafile 2024-09-16T16:26:22.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj468 td/osd-recovery-space/datafile 2024-09-16T16:26:22.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj469 td/osd-recovery-space/datafile 2024-09-16T16:26:22.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj470 td/osd-recovery-space/datafile 2024-09-16T16:26:22.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj471 td/osd-recovery-space/datafile 2024-09-16T16:26:22.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj472 td/osd-recovery-space/datafile 2024-09-16T16:26:22.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj473 td/osd-recovery-space/datafile 2024-09-16T16:26:22.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:22.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj474 td/osd-recovery-space/datafile 2024-09-16T16:26:23.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj475 td/osd-recovery-space/datafile 2024-09-16T16:26:23.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj476 td/osd-recovery-space/datafile 2024-09-16T16:26:23.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj477 td/osd-recovery-space/datafile 2024-09-16T16:26:23.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj478 td/osd-recovery-space/datafile 2024-09-16T16:26:23.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj479 td/osd-recovery-space/datafile 2024-09-16T16:26:23.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj480 td/osd-recovery-space/datafile 2024-09-16T16:26:23.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj481 td/osd-recovery-space/datafile 2024-09-16T16:26:23.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj482 td/osd-recovery-space/datafile 2024-09-16T16:26:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj483 td/osd-recovery-space/datafile 2024-09-16T16:26:23.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj484 td/osd-recovery-space/datafile 2024-09-16T16:26:23.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj485 td/osd-recovery-space/datafile 2024-09-16T16:26:23.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj486 td/osd-recovery-space/datafile 2024-09-16T16:26:23.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj487 td/osd-recovery-space/datafile 2024-09-16T16:26:23.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:23.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj488 td/osd-recovery-space/datafile 2024-09-16T16:26:24.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj489 td/osd-recovery-space/datafile 2024-09-16T16:26:24.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj490 td/osd-recovery-space/datafile 2024-09-16T16:26:24.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj491 td/osd-recovery-space/datafile 2024-09-16T16:26:24.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj492 td/osd-recovery-space/datafile 2024-09-16T16:26:24.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj493 td/osd-recovery-space/datafile 2024-09-16T16:26:24.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj494 td/osd-recovery-space/datafile 2024-09-16T16:26:24.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj495 td/osd-recovery-space/datafile 2024-09-16T16:26:24.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj496 td/osd-recovery-space/datafile 2024-09-16T16:26:24.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj497 td/osd-recovery-space/datafile 2024-09-16T16:26:24.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj498 td/osd-recovery-space/datafile 2024-09-16T16:26:24.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj499 td/osd-recovery-space/datafile 2024-09-16T16:26:24.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj500 td/osd-recovery-space/datafile 2024-09-16T16:26:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj501 td/osd-recovery-space/datafile 2024-09-16T16:26:25.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj502 td/osd-recovery-space/datafile 2024-09-16T16:26:25.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj503 td/osd-recovery-space/datafile 2024-09-16T16:26:25.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj504 td/osd-recovery-space/datafile 2024-09-16T16:26:25.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj505 td/osd-recovery-space/datafile 2024-09-16T16:26:25.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj506 td/osd-recovery-space/datafile 2024-09-16T16:26:25.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj507 td/osd-recovery-space/datafile 2024-09-16T16:26:25.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj508 td/osd-recovery-space/datafile 2024-09-16T16:26:25.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj509 td/osd-recovery-space/datafile 2024-09-16T16:26:25.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj510 td/osd-recovery-space/datafile 2024-09-16T16:26:25.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj511 td/osd-recovery-space/datafile 2024-09-16T16:26:25.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj512 td/osd-recovery-space/datafile 2024-09-16T16:26:25.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:25.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj513 td/osd-recovery-space/datafile 2024-09-16T16:26:26.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj514 td/osd-recovery-space/datafile 2024-09-16T16:26:26.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj515 td/osd-recovery-space/datafile 2024-09-16T16:26:26.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj516 td/osd-recovery-space/datafile 2024-09-16T16:26:26.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj517 td/osd-recovery-space/datafile 2024-09-16T16:26:26.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj518 td/osd-recovery-space/datafile 2024-09-16T16:26:26.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj519 td/osd-recovery-space/datafile 2024-09-16T16:26:26.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj520 td/osd-recovery-space/datafile 2024-09-16T16:26:26.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj521 td/osd-recovery-space/datafile 2024-09-16T16:26:26.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj522 td/osd-recovery-space/datafile 2024-09-16T16:26:26.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj523 td/osd-recovery-space/datafile 2024-09-16T16:26:26.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj524 td/osd-recovery-space/datafile 2024-09-16T16:26:26.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:26.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj525 td/osd-recovery-space/datafile 2024-09-16T16:26:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj526 td/osd-recovery-space/datafile 2024-09-16T16:26:27.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:27.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj527 td/osd-recovery-space/datafile 2024-09-16T16:26:27.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:27.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj528 td/osd-recovery-space/datafile 2024-09-16T16:26:27.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:27.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj529 td/osd-recovery-space/datafile 2024-09-16T16:26:27.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:27.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj530 td/osd-recovery-space/datafile 2024-09-16T16:26:27.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:27.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj531 td/osd-recovery-space/datafile 2024-09-16T16:26:27.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:27.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj532 td/osd-recovery-space/datafile 2024-09-16T16:26:27.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:27.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj533 td/osd-recovery-space/datafile 2024-09-16T16:26:27.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:27.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj534 td/osd-recovery-space/datafile 2024-09-16T16:26:27.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:27.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj535 td/osd-recovery-space/datafile 2024-09-16T16:26:28.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj536 td/osd-recovery-space/datafile 2024-09-16T16:26:28.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj537 td/osd-recovery-space/datafile 2024-09-16T16:26:28.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj538 td/osd-recovery-space/datafile 2024-09-16T16:26:28.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj539 td/osd-recovery-space/datafile 2024-09-16T16:26:28.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj540 td/osd-recovery-space/datafile 2024-09-16T16:26:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj541 td/osd-recovery-space/datafile 2024-09-16T16:26:28.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj542 td/osd-recovery-space/datafile 2024-09-16T16:26:28.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj543 td/osd-recovery-space/datafile 2024-09-16T16:26:28.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj544 td/osd-recovery-space/datafile 2024-09-16T16:26:28.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj545 td/osd-recovery-space/datafile 2024-09-16T16:26:28.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj546 td/osd-recovery-space/datafile 2024-09-16T16:26:28.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:28.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj547 td/osd-recovery-space/datafile 2024-09-16T16:26:29.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj548 td/osd-recovery-space/datafile 2024-09-16T16:26:29.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj549 td/osd-recovery-space/datafile 2024-09-16T16:26:29.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj550 td/osd-recovery-space/datafile 2024-09-16T16:26:29.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj551 td/osd-recovery-space/datafile 2024-09-16T16:26:29.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj552 td/osd-recovery-space/datafile 2024-09-16T16:26:29.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj553 td/osd-recovery-space/datafile 2024-09-16T16:26:29.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj554 td/osd-recovery-space/datafile 2024-09-16T16:26:29.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj555 td/osd-recovery-space/datafile 2024-09-16T16:26:29.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj556 td/osd-recovery-space/datafile 2024-09-16T16:26:29.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj557 td/osd-recovery-space/datafile 2024-09-16T16:26:29.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj558 td/osd-recovery-space/datafile 2024-09-16T16:26:29.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj559 td/osd-recovery-space/datafile 2024-09-16T16:26:29.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj560 td/osd-recovery-space/datafile 2024-09-16T16:26:29.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:29.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj561 td/osd-recovery-space/datafile 2024-09-16T16:26:30.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj562 td/osd-recovery-space/datafile 2024-09-16T16:26:30.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj563 td/osd-recovery-space/datafile 2024-09-16T16:26:30.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj564 td/osd-recovery-space/datafile 2024-09-16T16:26:30.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj565 td/osd-recovery-space/datafile 2024-09-16T16:26:30.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj566 td/osd-recovery-space/datafile 2024-09-16T16:26:30.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj567 td/osd-recovery-space/datafile 2024-09-16T16:26:30.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj568 td/osd-recovery-space/datafile 2024-09-16T16:26:30.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj569 td/osd-recovery-space/datafile 2024-09-16T16:26:30.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj570 td/osd-recovery-space/datafile 2024-09-16T16:26:30.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj571 td/osd-recovery-space/datafile 2024-09-16T16:26:30.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj572 td/osd-recovery-space/datafile 2024-09-16T16:26:30.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj573 td/osd-recovery-space/datafile 2024-09-16T16:26:30.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj574 td/osd-recovery-space/datafile 2024-09-16T16:26:30.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:30.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj575 td/osd-recovery-space/datafile 2024-09-16T16:26:31.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj576 td/osd-recovery-space/datafile 2024-09-16T16:26:31.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj577 td/osd-recovery-space/datafile 2024-09-16T16:26:31.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj578 td/osd-recovery-space/datafile 2024-09-16T16:26:31.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj579 td/osd-recovery-space/datafile 2024-09-16T16:26:31.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj580 td/osd-recovery-space/datafile 2024-09-16T16:26:31.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj581 td/osd-recovery-space/datafile 2024-09-16T16:26:31.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj582 td/osd-recovery-space/datafile 2024-09-16T16:26:31.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj583 td/osd-recovery-space/datafile 2024-09-16T16:26:31.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj584 td/osd-recovery-space/datafile 2024-09-16T16:26:31.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj585 td/osd-recovery-space/datafile 2024-09-16T16:26:31.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj586 td/osd-recovery-space/datafile 2024-09-16T16:26:31.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj587 td/osd-recovery-space/datafile 2024-09-16T16:26:31.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj588 td/osd-recovery-space/datafile 2024-09-16T16:26:31.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj589 td/osd-recovery-space/datafile 2024-09-16T16:26:31.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:31.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj590 td/osd-recovery-space/datafile 2024-09-16T16:26:32.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:32.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj591 td/osd-recovery-space/datafile 2024-09-16T16:26:32.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:32.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj592 td/osd-recovery-space/datafile 2024-09-16T16:26:32.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:32.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj593 td/osd-recovery-space/datafile 2024-09-16T16:26:32.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:32.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj594 td/osd-recovery-space/datafile 2024-09-16T16:26:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:32.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj595 td/osd-recovery-space/datafile 2024-09-16T16:26:32.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:32.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj596 td/osd-recovery-space/datafile 2024-09-16T16:26:32.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:32.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj597 td/osd-recovery-space/datafile 2024-09-16T16:26:32.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:32.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj598 td/osd-recovery-space/datafile 2024-09-16T16:26:32.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:32.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj599 td/osd-recovery-space/datafile 2024-09-16T16:26:32.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-16T16:26:32.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj600 td/osd-recovery-space/datafile 2024-09-16T16:26:32.923 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: expr 2 - 1 2024-09-16T16:26:32.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: seq 0 1 2024-09-16T16:26:32.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T16:26:32.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.0 injectargs '--fake_statfs_for_testing 3686400' 2024-09-16T16:26:33.072 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T16:26:33.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T16:26:33.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.1 injectargs '--fake_statfs_for_testing 3686400' 2024-09-16T16:26:33.231 INFO:tasks.workunit.client.0.smithi039.stdout:{} 2024-09-16T16:26:33.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: sleep 5 2024-09-16T16:26:38.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:126: TEST_recovery_test_simple: ceph pg dump pgs 2024-09-16T16:26:38.652 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T16:26:38.653 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 600 0 0 0 0 3072000 0 0 600 0 600 active+clean 2024-09-16T16:25:34.660099+0000 23'600 23:617 [1] 1 [1] 1 0'0 2024-09-16T16:25:32.205968+0000 0'0 2024-09-16T16:25:32.205968+0000 0 0 periodic scrub scheduled @ 2024-09-17T19:01:46.463997+0000 0 0 2024-09-16T16:26:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:26:38.653 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T16:26:38.653 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:26:38.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: seq 1 1 2024-09-16T16:26:38.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-09-16T16:26:38.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:130: TEST_recovery_test_simple: ceph osd pool set test1 size 2 2024-09-16T16:26:39.404 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-09-16T16:26:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:134: TEST_recovery_test_simple: wait_for_recovery_toofull 30 2024-09-16T16:26:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:77: wait_for_recovery_toofull: local timeout=30 2024-09-16T16:26:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:78: wait_for_recovery_toofull: wait_for_state recovery_toofull 30 2024-09-16T16:26:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:52: wait_for_state: local state=recovery_toofull 2024-09-16T16:26:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: local cur_in_state 2024-09-16T16:26:39.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: get_timeout_delays 30 5 2024-09-16T16:26:39.427 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:26:39.428 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:26:39.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:26:39.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:26:39.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:26:39.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: delays=('5' '10' '15') 2024-09-16T16:26:39.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: local -a delays 2024-09-16T16:26:39.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:55: wait_for_state: local -i loop=0 2024-09-16T16:26:39.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: flush_pg_stats 2024-09-16T16:26:39.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:26:39.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:26:40.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:26:40.080 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-09-16T16:26:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:26:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:26:40.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:26:40.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836512 2024-09-16T16:26:40.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836512 2024-09-16T16:26:40.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836512' 2024-09-16T16:26:40.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:26:40.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:26:40.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672986 2024-09-16T16:26:40.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672986 2024-09-16T16:26:40.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836512 1-42949672986' 2024-09-16T16:26:40.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:26:40.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836512 2024-09-16T16:26:40.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:26:40.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:26:40.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836512 2024-09-16T16:26:40.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:26:40.406 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836512 2024-09-16T16:26:40.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836512 2024-09-16T16:26:40.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-16T16:26:40.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:26:40.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836510 -lt 21474836512 2024-09-16T16:26:40.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:26:41.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:26:41.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:26:42.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-16T16:26:42.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:26:42.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672986 2024-09-16T16:26:42.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:26:42.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:26:42.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672986 2024-09-16T16:26:42.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:26:42.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672986 2024-09-16T16:26:42.253 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672986 2024-09-16T16:26:42.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-09-16T16:26:42.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:26:42.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672986 -lt 42949672986 2024-09-16T16:26:42.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: get_num_pgs 2024-09-16T16:26:42.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:26:42.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:26:43.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: test 1 == 0 2024-09-16T16:26:43.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: true 2024-09-16T16:26:43.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: get_num_in_state recovery_toofull 2024-09-16T16:26:43.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local state=recovery_toofull 2024-09-16T16:26:43.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: local expression 2024-09-16T16:26:43.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2024-09-16T16:26:43.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: ceph --format json pg dump pgs 2024-09-16T16:26:43.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:47: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2024-09-16T16:26:43.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: cur_in_state=1 2024-09-16T16:26:43.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: test 1 -gt 0 2024-09-16T16:26:43.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: break 2024-09-16T16:26:43.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:72: wait_for_state: return 0 2024-09-16T16:26:43.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: ERRORS=0 2024-09-16T16:26:43.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: ceph pg dump pgs 2024-09-16T16:26:43.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: grep +recovery_toofull 2024-09-16T16:26:43.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: wc -l 2024-09-16T16:26:44.042 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:26:44.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: '[' 1 '!=' 1 ']' 2024-09-16T16:26:44.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:143: TEST_recovery_test_simple: ceph pg dump pgs 2024-09-16T16:26:44.453 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T16:26:44.453 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 600 0 600 0 0 3072000 0 0 600 0 600 active+recovery_toofull+undersized+degraded+remapped 2024-09-16T16:26:39.957680+0000 23'600 27:628 [1,0] 1 [1] 1 0'0 2024-09-16T16:25:32.205968+0000 0'0 2024-09-16T16:25:32.205968+0000 0 0 periodic scrub scheduled @ 2024-09-18T02:14:52.358506+0000 0 0 2024-09-16T16:26:44.453 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:26:44.453 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T16:26:44.454 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:26:44.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:144: TEST_recovery_test_simple: ceph status 2024-09-16T16:26:44.992 INFO:tasks.workunit.client.0.smithi039.stdout: cluster: 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: id: f5249a10-5ed5-4225-b027-aef69043c2f6 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: health: HEALTH_ERR 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: 1 full osd(s) 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: Degraded data redundancy: 600/1200 objects degraded (50.000%), 1 pg degraded 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: Full OSDs blocking recovery: 1 pg recovery_toofull 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: 1 pool(s) do not have an application enabled 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: 1 pool(s) full 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: OSD count 2 < osd_pool_default_size 3 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: services: 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: mon: 1 daemons, quorum a (age 2m) 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: mgr: x(active, since 2m) 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: osd: 2 osds: 2 up (since 75s), 2 in (since 109s); 1 remapped pgs 2024-09-16T16:26:44.993 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:26:44.994 INFO:tasks.workunit.client.0.smithi039.stdout: data: 2024-09-16T16:26:44.994 INFO:tasks.workunit.client.0.smithi039.stdout: pools: 1 pools, 1 pgs 2024-09-16T16:26:44.994 INFO:tasks.workunit.client.0.smithi039.stdout: objects: 600 objects, 2.9 MiB 2024-09-16T16:26:44.994 INFO:tasks.workunit.client.0.smithi039.stdout: usage: 2.9 MiB used, 4.1 MiB / 7.0 MiB avail 2024-09-16T16:26:44.994 INFO:tasks.workunit.client.0.smithi039.stdout: pgs: 600/1200 objects degraded (50.000%) 2024-09-16T16:26:44.994 INFO:tasks.workunit.client.0.smithi039.stdout: 1 active+recovery_toofull+undersized+degraded+remapped 2024-09-16T16:26:44.994 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T16:26:45.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: ceph status --format=json-pretty 2024-09-16T16:26:45.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.severity td/osd-recovery-space/stat.json 2024-09-16T16:26:45.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: eval 'SEV="HEALTH_ERR"' 2024-09-16T16:26:45.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: SEV=HEALTH_ERR 2024-09-16T16:26:45.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:148: TEST_recovery_test_simple: '[' HEALTH_ERR '!=' HEALTH_ERR ']' 2024-09-16T16:26:45.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.summary.message td/osd-recovery-space/stat.json 2024-09-16T16:26:45.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: eval 'MSG="Full' OSDs blocking recovery: 1 pg 'recovery_toofull"' 2024-09-16T16:26:45.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: MSG='Full OSDs blocking recovery: 1 pg recovery_toofull' 2024-09-16T16:26:45.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:153: TEST_recovery_test_simple: '[' 'Full OSDs blocking recovery: 1 pg recovery_toofull' '!=' 'Full OSDs blocking recovery: 1 pg recovery_toofull' ']' 2024-09-16T16:26:45.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:157: TEST_recovery_test_simple: rm -f td/osd-recovery-space/stat.json 2024-09-16T16:26:45.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:159: TEST_recovery_test_simple: '[' 0 '!=' 0 ']' 2024-09-16T16:26:45.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:164: TEST_recovery_test_simple: seq 1 1 2024-09-16T16:26:45.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:164: TEST_recovery_test_simple: for i in $(seq 1 $pools) 2024-09-16T16:26:45.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:166: TEST_recovery_test_simple: delete_pool test1 2024-09-16T16:26:45.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-09-16T16:26:45.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-09-16T16:26:46.366 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test1' does not exist 2024-09-16T16:26:46.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:168: TEST_recovery_test_simple: kill_daemons td/osd-recovery-space 2024-09-16T16:26:46.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:26:46.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:26:46.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:26:46.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:26:46.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:26:51.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:26:51.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:37: run: teardown td/osd-recovery-space 2024-09-16T16:26:51.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-09-16T16:26:51.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:26:51.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-09-16T16:26:51.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:26:51.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:26:51.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:26:51.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:26:51.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:26:51.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:26:51.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:26:51.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:26:51.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:26:51.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:26:51.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:26:51.713 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:26:51.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:26:51.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:26:51.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:26:51.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:26:51.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:26:51.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:26:51.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-09-16T16:26:51.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:26:51.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:26:51.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258350 2024-09-16T16:26:51.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.258350 2024-09-16T16:26:51.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:26:51.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:26:51.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T16:26:51.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-recovery-space 0 2024-09-16T16:26:51.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-09-16T16:26:51.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T16:26:51.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-09-16T16:26:51.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:26:51.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:26:51.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:26:51.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:26:51.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:26:51.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:26:51.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:26:51.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:26:51.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:26:51.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:26:51.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:26:51.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:26:51.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:26:51.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:26:51.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:26:51.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:26:51.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:26:51.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T16:26:51.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-09-16T16:26:51.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:26:51.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:26:51.770 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258350 2024-09-16T16:26:51.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.258350 2024-09-16T16:26:51.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:26:51.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:26:51.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T16:26:51.772 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T16:26:51.772 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T16:26:51.783 INFO:tasks.workunit:Running workunit osd/osd-recovery-stats.sh... 2024-09-16T16:26:51.783 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-recovery-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh 2024-09-16T16:26:51.835 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T16:26:51.842 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T16:26:51.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T16:26:51.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T16:26:51.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T16:26:51.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T16:26:51.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T16:26:51.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T16:26:51.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T16:26:51.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T16:26:51.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-recovery-stats 2024-09-16T16:26:51.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:21: run: local dir=td/osd-recovery-stats 2024-09-16T16:26:51.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:22: run: shift 2024-09-16T16:26:51.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: export CEPH_MON=127.0.0.1:7115 2024-09-16T16:26:51.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: CEPH_MON=127.0.0.1:7115 2024-09-16T16:26:51.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:26: run: export CEPH_ARGS 2024-09-16T16:26:51.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: uuidgen 2024-09-16T16:26:51.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: CEPH_ARGS+='--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none ' 2024-09-16T16:26:51.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-16T16:26:51.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:30: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:26:51.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: export margin=10 2024-09-16T16:26:51.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: margin=10 2024-09-16T16:26:51.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: export objects=200 2024-09-16T16:26:51.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: objects=200 2024-09-16T16:26:51.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: export poolname=test 2024-09-16T16:26:51.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: poolname=test 2024-09-16T16:26:51.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: set 2024-09-16T16:26:51.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T16:26:51.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: local 'funcs=TEST_recovery_erasure_out1 2024-09-16T16:26:51.847 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_recovery_erasure_remapped 2024-09-16T16:26:51.847 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_recovery_multi 2024-09-16T16:26:51.847 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_recovery_replicated_out1 2024-09-16T16:26:51.847 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_recovery_sizedown 2024-09-16T16:26:51.847 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_recovery_sizeup 2024-09-16T16:26:51.847 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_recovery_undersized' 2024-09-16T16:26:51.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-16T16:26:51.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-16T16:26:51.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-16T16:26:51.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-16T16:26:51.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:26:51.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:26:51.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:26:51.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:26:51.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:26:51.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:26:51.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:26:51.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:26:51.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:26:51.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:26:51.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:26:51.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:26:51.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:26:51.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:26:51.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:26:51.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:26:51.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:26:51.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:26:51.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:26:51.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:26:51.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:26:51.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:26:51.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:26:51.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:26:51.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:26:51.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:26:51.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:26:51.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:26:51.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-16T16:26:51.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:26:51.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:26:51.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:26:51.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274941 2024-09-16T16:26:51.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:26:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T16:26:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T16:26:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:26:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-16T16:26:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_out1 td/osd-recovery-stats 2024-09-16T16:26:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:173: TEST_recovery_erasure_out1: local dir=td/osd-recovery-stats 2024-09-16T16:26:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:175: TEST_recovery_erasure_out1: do_recovery_out1 td/osd-recovery-stats erasure 2024-09-16T16:26:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-09-16T16:26:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-09-16T16:26:51.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=erasure 2024-09-16T16:26:51.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-09-16T16:26:51.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-16T16:26:51.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:26:51.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:26:51.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:26:51.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-16T16:26:51.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-16T16:26:52.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:26:52.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:26:52.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:26:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:26:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:26:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:26:52.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:26:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:26:52.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:26:52.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:26:52.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:26:52.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:26:52.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:26:52.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:26:52.435 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:26:52.435 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:26:52.435 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:26:52.435 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:26:52.435 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:26:52.435 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:26:52.435 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:26:52.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:26:52.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get fsid 2024-09-16T16:26:52.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:26:52.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:26:52.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:26:52.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:26:52.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:26:52.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:26:52.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:26:52.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:26:52.557 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:26:52.557 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:26:52.557 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:26:52.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:26:52.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:26:52.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get mon_host 2024-09-16T16:26:52.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-09-16T16:26:52.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-16T16:26:52.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:26:52.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:26:52.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:26:52.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-16T16:26:52.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:26:53.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:26:53.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:26:53.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:26:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:26:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:26:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:26:53.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:26:53.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:26:53.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:26:53.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-09-16T16:26:53.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:26:53.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:26:53.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:26:53.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:26:53.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:26:53.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:26:53.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:26:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:26:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:26:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:26:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-16T16:26:53.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:26:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30e5eb4e-83bf-468c-9391-ee58cb5336e7 2024-09-16T16:26:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 30e5eb4e-83bf-468c-9391-ee58cb5336e7' 2024-09-16T16:26:53.632 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 30e5eb4e-83bf-468c-9391-ee58cb5336e7 2024-09-16T16:26:53.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:26:53.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNXOhm/tipJhAAEFyIqawrF1ID9PXLzqpBqg== 2024-09-16T16:26:53.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNXOhm/tipJhAAEFyIqawrF1ID9PXLzqpBqg=="}' 2024-09-16T16:26:53.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30e5eb4e-83bf-468c-9391-ee58cb5336e7 -i td/osd-recovery-stats/0/new.json 2024-09-16T16:26:53.959 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:26:53.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-16T16:26:53.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNXOhm/tipJhAAEFyIqawrF1ID9PXLzqpBqg== --osd-uuid 30e5eb4e-83bf-468c-9391-ee58cb5336e7 2024-09-16T16:26:54.002 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:26:54.000+0000 7f98d0a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:26:54.004 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:26:54.000+0000 7f98d0a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:26:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:26:54.004+0000 7f98d0a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:26:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:26:54.004+0000 7f98d0a2b7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:26:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:26:54.004+0000 7f98d0a2b7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:26:54.006 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:26:54.004+0000 7f98d0a2b7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:26:54.036 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:26:54.032+0000 7f98d0a2b7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-16T16:27:00.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-16T16:27:00.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:27:00.269 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:27:00.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:27:00.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:27:00.839 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:27:00.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:27:00.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:27:00.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:27:00.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:27:00.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:27:00.871 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:00.867+0000 7f85870377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:27:00.873 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:00.867+0000 7f85870377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:27:00.874 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:00.871+0000 7f85870377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:27:01.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:27:01.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:27:01.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:27:01.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:27:01.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:27:01.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:01.319 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:27:01.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:27:01.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:01.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:01.434 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:01.431+0000 7f85870377c0 -1 Falling back to public interface 2024-09-16T16:27:01.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:02.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:02.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:02.758 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:27:02.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:27:02.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:02.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:03.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:03.956 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:03.951+0000 7f85870377c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:27:04.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:04.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:04.188 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:27:04.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:27:04.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:04.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:04.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:05.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:05.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:05.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:27:05.852 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:27:05.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:05.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:06.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:07.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:07.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:07.315 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:27:07.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:27:07.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:07.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:07.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:08.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:08.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:08.839 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:27:08.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:27:08.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:08.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:09.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:10.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:10.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:10.292 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:27:10.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:27:10.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:10.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:10.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:11.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:11.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:11.827 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:27:11.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:27:11.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:11.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:12.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:13.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:13.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:13.283 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:27:13.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:27:13.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:13.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:13.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:14.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:14.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:14.841 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:27:14.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:27:14.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:14.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:15.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:16.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:16.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:16.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:27:16.299 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:27:16.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:16.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:17.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:18.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:18.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:18.040 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:27:18.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:27:18.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:18.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:18.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:19.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:19.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:19.722 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:27:19.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:27:19.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:19.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:20.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:21.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:21.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:21.693 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:27:21.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:27:21.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:21.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:22.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:23.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:23.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:23.534 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:27:23.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:27:23.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:23.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:24.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:25.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:25.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:25.601 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:27:25.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:27:25.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:25.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:26.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:27.241 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:27:27.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:27:27.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:27.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:27:27.663 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1901836150,v1:127.0.0.1:6803/1901836150] [v2:127.0.0.1:6804/1901836150,v1:127.0.0.1:6805/1901836150] exists,up 30e5eb4e-83bf-468c-9391-ee58cb5336e7 2024-09-16T16:27:27.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:27:27.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:27:27.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:27:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:27:27.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:27:27.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:27:27.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:27:27.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:27:27.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:27:27.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:27:27.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-16T16:27:27.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:27:27.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a9c51052-ebec-4e3e-9294-90c12cf0b6a7 2024-09-16T16:27:27.669 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 a9c51052-ebec-4e3e-9294-90c12cf0b6a7 2024-09-16T16:27:27.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a9c51052-ebec-4e3e-9294-90c12cf0b6a7' 2024-09-16T16:27:27.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:27:27.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvXOhmDDhAKRAASohVeX8fTfS2wgSg0dD/zQ== 2024-09-16T16:27:27.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvXOhmDDhAKRAASohVeX8fTfS2wgSg0dD/zQ=="}' 2024-09-16T16:27:27.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a9c51052-ebec-4e3e-9294-90c12cf0b6a7 -i td/osd-recovery-stats/1/new.json 2024-09-16T16:27:28.276 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:27:28.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-16T16:27:28.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvXOhmDDhAKRAASohVeX8fTfS2wgSg0dD/zQ== --osd-uuid a9c51052-ebec-4e3e-9294-90c12cf0b6a7 2024-09-16T16:27:28.332 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:28.327+0000 7f1e7d53b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:27:28.357 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:28.355+0000 7f1e7d53b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:27:28.359 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:28.355+0000 7f1e7d53b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:27:28.359 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:28.355+0000 7f1e7d53b7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:27:28.359 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:28.355+0000 7f1e7d53b7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:27:28.359 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:28.355+0000 7f1e7d53b7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:27:28.449 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:28.447+0000 7f1e7d53b7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-16T16:27:34.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-16T16:27:34.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:27:34.489 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:27:34.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:27:34.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:27:35.049 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:27:35.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:27:35.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:27:35.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:27:35.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:27:35.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:27:35.082 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:35.083+0000 7fcc666117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:27:35.098 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:35.099+0000 7fcc666117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:27:35.100 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:35.099+0000 7fcc666117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:27:35.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:27:35.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:27:35.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:27:35.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:27:35.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:27:35.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:35.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:27:35.504 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:27:35.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:35.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:35.659 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:35.659+0000 7fcc666117c0 -1 Falling back to public interface 2024-09-16T16:27:35.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:36.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:36.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:36.981 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:27:36.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:27:36.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:36.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:37.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:38.235 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:27:38.235+0000 7fcc666117c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:27:38.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:38.461 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:27:38.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:38.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:27:38.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:38.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:39.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:40.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:40.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:40.068 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:27:40.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:27:40.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:40.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:40.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:41.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:41.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:41.556 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:27:41.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:27:41.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:41.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:42.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:43.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:43.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:43.130 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:27:43.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:27:43.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:43.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:43.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:44.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:44.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:44.629 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:27:44.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:27:44.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:44.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:45.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:46.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:46.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:46.089 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:27:46.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:27:46.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:46.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:46.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:47.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:47.530 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:27:47.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:47.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:27:47.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:47.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:48.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:49.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:49.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:49.068 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:27:49.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:27:49.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:49.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:49.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:50.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:50.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:50.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:27:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:27:50.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:50.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:51.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:52.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:52.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:52.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:27:52.115 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:27:52.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:52.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:53.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:54.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:54.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:54.379 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:27:54.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:27:54.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:54.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:55.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:56.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:56.644 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:27:56.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:56.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:27:56.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:56.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:57.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:27:58.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:27:58.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:27:58.606 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:27:58.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:27:58.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:27:58.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:27:59.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:00.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:00.541 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:28:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:28:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:28:01.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:02.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:02.389 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:28:02.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:02.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:28:02.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:02.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:28:02.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:03.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:03.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:03.917 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:28:03.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:28:03.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:03.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:28:04.447 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2313709847,v1:127.0.0.1:6811/2313709847] [v2:127.0.0.1:6812/2313709847,v1:127.0.0.1:6813/2313709847] exists,up a9c51052-ebec-4e3e-9294-90c12cf0b6a7 2024-09-16T16:28:04.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:28:04.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-16T16:28:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:28:04.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:28:04.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:28:04.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:28:04.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:28:04.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:28:04.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:28:04.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-16T16:28:04.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:28:04.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=06606c19-8d54-40d5-b71f-2230ef61db14 2024-09-16T16:28:04.456 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 06606c19-8d54-40d5-b71f-2230ef61db14 2024-09-16T16:28:04.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 06606c19-8d54-40d5-b71f-2230ef61db14' 2024-09-16T16:28:04.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:28:04.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUXOhmmo9mHBAAVJv8fS+p/fRqRVncRyHF5g== 2024-09-16T16:28:04.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUXOhmmo9mHBAAVJv8fS+p/fRqRVncRyHF5g=="}' 2024-09-16T16:28:04.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 06606c19-8d54-40d5-b71f-2230ef61db14 -i td/osd-recovery-stats/2/new.json 2024-09-16T16:28:05.081 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:28:05.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-16T16:28:05.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUXOhmmo9mHBAAVJv8fS+p/fRqRVncRyHF5g== --osd-uuid 06606c19-8d54-40d5-b71f-2230ef61db14 2024-09-16T16:28:05.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:05.138+0000 7fcf236bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:05.162 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:05.162+0000 7fcf236bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:05.164 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:05.162+0000 7fcf236bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:05.164 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:05.162+0000 7fcf236bb7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:28:05.165 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:05.162+0000 7fcf236bb7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:28:05.165 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:05.162+0000 7fcf236bb7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:28:05.263 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:05.262+0000 7fcf236bb7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-16T16:28:11.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-16T16:28:11.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:28:11.029 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T16:28:11.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T16:28:11.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:28:11.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T16:28:11.592 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T16:28:11.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:28:11.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:28:11.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:28:11.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:28:11.617 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:11.614+0000 7f98112e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:11.653 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:11.650+0000 7f98112e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:11.654 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:11.654+0000 7f98112e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:12.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T16:28:12.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:28:12.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T16:28:12.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:28:12.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:28:12.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:12.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:28:12.040 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:28:12.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:12.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:12.227 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:12.226+0000 7f98112e07c0 -1 Falling back to public interface 2024-09-16T16:28:12.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:13.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:13.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:13.453 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:28:13.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:28:13.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:13.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:13.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:14.587 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:14.586+0000 7f98112e07c0 -1 osd.2 0 log_to_monitors true 2024-09-16T16:28:14.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:14.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:14.903 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:28:14.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:28:14.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:14.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:15.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:16.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:16.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:16.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:28:16.509 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:28:16.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:16.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:17.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:18.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:18.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:18.014 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:28:18.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:28:18.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:18.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:18.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:19.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:19.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:19.579 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:28:19.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:28:19.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:19.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:20.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:21.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:21.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:21.028 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:28:21.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:28:21.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:21.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:21.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:22.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:22.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:22.553 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:28:22.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:28:22.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:22.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:23.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:24.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:24.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:24.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:28:24.172 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:28:24.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:24.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:24.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:25.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:25.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:25.602 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:28:25.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:28:25.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:25.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:26.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:27.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:27.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:27.079 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:28:27.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:28:27.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:27.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:27.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:28.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:28.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:28.640 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:28:28.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:28:28.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:28.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:29.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:30.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:30.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:30.777 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:28:30.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:28:30.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:30.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:31.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:32.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:32.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:32.407 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:28:32.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:28:32.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:32.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:33.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:34.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:34.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:34.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:28:34.442 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:28:34.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:34.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:34.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:35.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:35.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:35.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:28:35.868 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:28:35.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:35.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:36.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:37.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:37.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:37.396 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:28:37.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:28:37.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:37.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:37.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:38.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:38.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:38.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:28:38.952 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:28:38.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:38.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:28:39.382 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1350243066,v1:127.0.0.1:6819/1350243066] [v2:127.0.0.1:6820/1350243066,v1:127.0.0.1:6821/1350243066] exists,up 06606c19-8d54-40d5-b71f-2230ef61db14 2024-09-16T16:28:39.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:28:39.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:28:39.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:28:39.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-09-16T16:28:39.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:28:39.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:28:39.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:28:39.385 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:28:39.386 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:28:39.386 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:28:39.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:28:39.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:28:39.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:28:39.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:28:39.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:28:39.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:28:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:28:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:28:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:28:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:28:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:28:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:28:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:28:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-16T16:28:39.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:28:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0428ead4-642e-45fd-b748-352731ffed0a 2024-09-16T16:28:39.388 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 0428ead4-642e-45fd-b748-352731ffed0a 2024-09-16T16:28:39.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0428ead4-642e-45fd-b748-352731ffed0a' 2024-09-16T16:28:39.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:28:39.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3XOhmwIGLGBAAUFAzyB9hvPQTOfhUvIwo1Q== 2024-09-16T16:28:39.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3XOhmwIGLGBAAUFAzyB9hvPQTOfhUvIwo1Q=="}' 2024-09-16T16:28:39.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0428ead4-642e-45fd-b748-352731ffed0a -i td/osd-recovery-stats/3/new.json 2024-09-16T16:28:40.065 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:28:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-16T16:28:40.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3XOhmwIGLGBAAUFAzyB9hvPQTOfhUvIwo1Q== --osd-uuid 0428ead4-642e-45fd-b748-352731ffed0a 2024-09-16T16:28:40.109 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:40.105+0000 7faf0f3b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:40.137 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:40.133+0000 7faf0f3b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:40.138 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:40.137+0000 7faf0f3b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:40.139 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:40.137+0000 7faf0f3b57c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:28:40.139 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:40.137+0000 7faf0f3b57c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:28:40.139 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:40.137+0000 7faf0f3b57c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:28:40.229 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:40.225+0000 7faf0f3b57c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-16T16:28:46.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-16T16:28:46.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:28:46.304 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T16:28:46.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T16:28:46.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:28:46.859 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T16:28:46.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T16:28:46.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:28:46.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:28:46.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:28:46.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:28:46.885 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:46.881+0000 7f5a70a567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:46.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:46.893+0000 7f5a70a567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:46.896 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:46.893+0000 7f5a70a567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:28:47.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T16:28:47.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:28:47.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T16:28:47.261 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:28:47.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:28:47.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:28:47.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:47.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:28:47.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:47.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:28:47.475 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:47.473+0000 7f5a70a567c0 -1 Falling back to public interface 2024-09-16T16:28:47.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:48.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:48.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:48.745 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:28:48.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:28:48.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:48.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:28:49.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:50.110 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:28:50.109+0000 7f5a70a567c0 -1 osd.3 0 log_to_monitors true 2024-09-16T16:28:50.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:50.249 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:28:50.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:50.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:28:50.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:50.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:28:50.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:51.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:51.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:51.726 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:28:51.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:28:51.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:51.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:28:52.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:53.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:53.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:53.266 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:28:53.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:28:53.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:53.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:28:53.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:54.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:54.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:54.738 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:28:54.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:28:54.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:54.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:28:55.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:56.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:56.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:56.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:28:56.252 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:28:56.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:56.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:28:56.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:57.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:57.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:57.701 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:28:57.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:28:57.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:57.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:28:58.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:28:59.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:28:59.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:28:59.233 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:28:59.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:28:59.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:28:59.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:28:59.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:00.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:00.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:00.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:29:00.709 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:29:00.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:00.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:29:01.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:02.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:02.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:02.231 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:29:02.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:29:02.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:02.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:29:02.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:03.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:03.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:03.823 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:29:03.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:29:03.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:03.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:29:04.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:05.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:05.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:05.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:29:05.686 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:29:05.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:05.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:29:06.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:07.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:07.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:07.777 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:29:07.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:29:07.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:07.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:29:08.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:09.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:09.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:09.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:29:09.771 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:29:09.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:09.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:29:10.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:11.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:11.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:11.325 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:29:11.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:29:11.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:11.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:29:12.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:13.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:13.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:13.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:29:13.491 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:29:13.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:13.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:29:14.016 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1663872930,v1:127.0.0.1:6827/1663872930] [v2:127.0.0.1:6828/1663872930,v1:127.0.0.1:6829/1663872930] exists,up 0428ead4-642e-45fd-b748-352731ffed0a 2024-09-16T16:29:14.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:29:14.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:29:14.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:29:14.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:29:14.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:29:14.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:29:14.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:29:14.020 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:29:14.020 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:29:14.020 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:29:14.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:29:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-16T16:29:14.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:29:14.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cf0ffbc7-d0b6-4462-ac23-9ea03648d0d4 2024-09-16T16:29:14.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 cf0ffbc7-d0b6-4462-ac23-9ea03648d0d4' 2024-09-16T16:29:14.023 INFO:tasks.workunit.client.0.smithi039.stdout:add osd4 cf0ffbc7-d0b6-4462-ac23-9ea03648d0d4 2024-09-16T16:29:14.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:29:14.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDaXOhmqJWKAhAAdY/B7BVE61/tx37z3Okspw== 2024-09-16T16:29:14.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDaXOhmqJWKAhAAdY/B7BVE61/tx37z3Okspw=="}' 2024-09-16T16:29:14.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cf0ffbc7-d0b6-4462-ac23-9ea03648d0d4 -i td/osd-recovery-stats/4/new.json 2024-09-16T16:29:14.675 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:29:14.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-16T16:29:14.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaXOhmqJWKAhAAdY/B7BVE61/tx37z3Okspw== --osd-uuid cf0ffbc7-d0b6-4462-ac23-9ea03648d0d4 2024-09-16T16:29:14.732 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:14.729+0000 7fb6ee53d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:14.780 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:14.777+0000 7fb6ee53d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:14.782 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:14.781+0000 7fb6ee53d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:14.783 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:14.781+0000 7fb6ee53d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:29:14.783 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:14.781+0000 7fb6ee53d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:29:14.783 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:14.781+0000 7fb6ee53d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:29:14.864 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:14.861+0000 7fb6ee53d7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-16T16:29:20.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-16T16:29:20.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:29:20.468 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd4 key to auth repository 2024-09-16T16:29:20.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T16:29:20.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:29:21.046 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.4 2024-09-16T16:29:21.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T16:29:21.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:29:21.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:29:21.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:29:21.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:29:21.080 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:21.076+0000 7fd82ba897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:21.082 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:21.080+0000 7fd82ba897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:21.084 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:21.080+0000 7fd82ba897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T16:29:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:29:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T16:29:21.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:29:21.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:29:21.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:21.469 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:29:21.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:29:21.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:21.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:21.627 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:21.624+0000 7fd82ba897c0 -1 Falling back to public interface 2024-09-16T16:29:21.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:22.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:22.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:22.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:29:22.927 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:29:22.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:22.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:23.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:24.188 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:24.184+0000 7fd82ba897c0 -1 osd.4 0 log_to_monitors true 2024-09-16T16:29:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:24.358 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:29:24.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:29:24.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:24.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:24.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:25.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:25.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:25.808 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:29:25.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:29:25.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:25.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:26.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:27.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:27.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:27.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:29:27.283 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:29:27.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:27.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:27.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:28.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:28.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:28.705 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:29:28.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:29:28.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:28.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:29.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:30.246 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:29:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:29:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:30.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:30.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:31.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:31.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:31.695 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:29:31.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:29:31.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:31.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:32.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:33.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:33.159 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:29:33.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:29:33.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:33.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:33.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:34.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:34.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:34.575 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:29:34.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:29:34.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:34.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:35.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:36.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:36.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:36.151 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:29:36.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:29:36.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:36.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:36.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:37.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:37.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:37.751 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:29:37.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:29:37.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:37.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:38.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:39.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:39.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:39.611 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:29:39.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:29:39.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:39.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:40.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:41.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:41.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:41.516 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:29:41.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:29:41.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:41.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:42.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:43.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:43.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:43.349 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:29:43.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:29:43.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:43.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:44.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:45.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:45.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:45.238 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:29:45.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:29:45.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:45.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:46.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:47.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:47.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:47.431 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:29:47.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:29:47.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:47.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:29:47.974 INFO:tasks.workunit.client.0.smithi039.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/148028480,v1:127.0.0.1:6835/148028480] [v2:127.0.0.1:6836/148028480,v1:127.0.0.1:6837/148028480] exists,up cf0ffbc7-d0b6-4462-ac23-9ea03648d0d4 2024-09-16T16:29:47.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:29:47.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:29:47.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:29:47.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-09-16T16:29:47.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:29:47.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:29:47.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:29:47.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:29:47.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:29:47.978 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:29:47.978 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:29:47.978 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:29:47.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:29:47.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:29:47.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:29:47.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:29:47.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:29:47.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:29:47.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:29:47.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:29:47.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:29:47.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:29:47.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:29:47.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:29:47.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:29:47.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-16T16:29:47.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:29:47.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7b061cc7-e2e3-41dc-b695-aeef48c95b9c 2024-09-16T16:29:47.980 INFO:tasks.workunit.client.0.smithi039.stdout:add osd5 7b061cc7-e2e3-41dc-b695-aeef48c95b9c 2024-09-16T16:29:47.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 7b061cc7-e2e3-41dc-b695-aeef48c95b9c' 2024-09-16T16:29:47.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:29:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8XOhmc5ElABAADGKm6r3a9am66wfQCYe+7w== 2024-09-16T16:29:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8XOhmc5ElABAADGKm6r3a9am66wfQCYe+7w=="}' 2024-09-16T16:29:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7b061cc7-e2e3-41dc-b695-aeef48c95b9c -i td/osd-recovery-stats/5/new.json 2024-09-16T16:29:48.661 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:29:48.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-16T16:29:48.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8XOhmc5ElABAADGKm6r3a9am66wfQCYe+7w== --osd-uuid 7b061cc7-e2e3-41dc-b695-aeef48c95b9c 2024-09-16T16:29:48.723 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:48.720+0000 7f9de96377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:48.805 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:48.800+0000 7f9de96377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:48.807 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:48.804+0000 7f9de96377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:48.807 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:48.804+0000 7f9de96377c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:29:48.807 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:48.804+0000 7f9de96377c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:29:48.808 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:48.804+0000 7f9de96377c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:29:48.884 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:48.884+0000 7f9de96377c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-16T16:29:54.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-16T16:29:54.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:29:54.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T16:29:54.966 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd5 key to auth repository 2024-09-16T16:29:54.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:29:55.526 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.5 2024-09-16T16:29:55.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T16:29:55.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:29:55.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:29:55.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:29:55.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:29:55.559 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:55.556+0000 7feaa74437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:55.575 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:55.572+0000 7feaa74437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:55.576 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:55.572+0000 7feaa74437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:29:55.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T16:29:55.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:29:55.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T16:29:55.966 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:29:55.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:29:55.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:29:55.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:55.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:29:55.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:55.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:29:56.160 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:56.156+0000 7feaa74437c0 -1 Falling back to public interface 2024-09-16T16:29:56.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:57.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:57.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:57.415 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:29:57.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:29:57.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:57.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:29:57.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:29:58.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:29:58.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:29:58.846 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:29:58.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:29:58.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:29:58.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:29:58.913 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:29:58.908+0000 7feaa74437c0 -1 osd.5 0 log_to_monitors true 2024-09-16T16:29:59.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:00.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:00.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:00.315 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:30:00.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:30:00.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:00.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:00.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:01.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:01.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:01.916 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:30:01.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:30:01.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:01.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:02.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:03.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:03.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:03.394 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:30:03.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:30:03.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:03.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:03.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:04.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:04.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:04.942 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:30:04.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:30:04.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:04.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:05.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:06.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:06.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:06.425 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:30:06.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:30:06.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:06.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:06.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:07.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:07.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:07.982 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:30:07.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:30:07.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:07.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:08.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:09.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:09.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:09.451 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:30:09.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:30:09.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:09.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:09.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:10.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:10.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:10.950 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:30:10.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:30:10.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:10.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:11.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:12.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:12.423 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:30:12.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:12.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:30:12.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:12.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:12.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:13.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:13.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:13.888 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:30:13.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:30:13.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:13.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:14.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:15.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:15.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:15.795 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:30:15.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:30:15.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:15.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:16.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:17.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:17.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:17.654 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:30:17.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:30:17.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:17.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:18.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:19.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:19.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:19.469 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:30:19.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:30:19.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:19.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:20.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:21.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:21.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:21.286 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:30:21.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:30:21.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:21.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:22.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:23.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:23.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:23.499 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:30:23.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:30:23.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:23.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:24.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:30:25.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:30:25.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:30:25.026 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T16:30:25.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T16:30:25.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:30:25.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:30:25.443 INFO:tasks.workunit.client.0.smithi039.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3280642945,v1:127.0.0.1:6843/3280642945] [v2:127.0.0.1:6844/3280642945,v1:127.0.0.1:6845/3280642945] exists,up 7b061cc7-e2e3-41dc-b695-aeef48c95b9c 2024-09-16T16:30:25.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:30:25.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:30:25.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:30:25.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' erasure = erasure ']' 2024-09-16T16:30:25.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:132: do_recovery_out1: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T16:30:26.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:133: do_recovery_out1: create_pool test 1 1 erasure myprofile 2024-09-16T16:30:26.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-16T16:30:26.925 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T16:30:26.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:30:27.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-09-16T16:30:27.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:30:27.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:30:27.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:30:27.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:30:27.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:30:27.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:30:27.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:30:27.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:30:27.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:30:28.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:30:28.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:30:28.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:30:28.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:30:28.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:30:28.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:30:28.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:30:28.567 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:30:28.567 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:30:28.567 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:30:28.567 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T16:30:28.568 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T16:30:28.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:30:28.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:30:28.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:30:28.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836520 2024-09-16T16:30:28.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836520 2024-09-16T16:30:28.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520' 2024-09-16T16:30:28.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:30:28.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:30:28.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672998 2024-09-16T16:30:28.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672998 2024-09-16T16:30:28.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520 1-42949672998' 2024-09-16T16:30:28.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:30:28.882 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:30:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509468 2024-09-16T16:30:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509468 2024-09-16T16:30:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520 1-42949672998 2-64424509468' 2024-09-16T16:30:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:30:29.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:30:29.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345942 2024-09-16T16:30:29.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345942 2024-09-16T16:30:29.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520 1-42949672998 2-64424509468 3-85899345942' 2024-09-16T16:30:29.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:30:29.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:30:29.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182417 2024-09-16T16:30:29.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182417 2024-09-16T16:30:29.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520 1-42949672998 2-64424509468 3-85899345942 4-107374182417' 2024-09-16T16:30:29.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:30:29.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T16:30:29.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018888 2024-09-16T16:30:29.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018888 2024-09-16T16:30:29.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520 1-42949672998 2-64424509468 3-85899345942 4-107374182417 5-128849018888' 2024-09-16T16:30:29.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:30:29.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836520 2024-09-16T16:30:29.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:30:29.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:30:29.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836520 2024-09-16T16:30:29.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:30:29.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836520 2024-09-16T16:30:29.509 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836520 2024-09-16T16:30:29.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-09-16T16:30:29.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:30:29.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836518 -lt 21474836520 2024-09-16T16:30:29.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:30:30.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:30:30.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:30:31.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836520 -lt 21474836520 2024-09-16T16:30:31.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:30:31.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672998 2024-09-16T16:30:31.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:30:31.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:30:31.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:30:31.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672998 2024-09-16T16:30:31.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672998 2024-09-16T16:30:31.347 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672998 2024-09-16T16:30:31.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-09-16T16:30:31.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:30:31.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672999 -lt 42949672998 2024-09-16T16:30:31.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:30:31.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509468 2024-09-16T16:30:31.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:30:31.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:30:31.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509468 2024-09-16T16:30:31.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:30:31.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509468 2024-09-16T16:30:31.756 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509468 2024-09-16T16:30:31.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-09-16T16:30:31.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:30:32.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509469 -lt 64424509468 2024-09-16T16:30:32.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:30:32.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345942 2024-09-16T16:30:32.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:30:32.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:30:32.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345942 2024-09-16T16:30:32.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:30:32.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345942 2024-09-16T16:30:32.175 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345942 2024-09-16T16:30:32.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2024-09-16T16:30:32.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:30:32.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345942 -lt 85899345942 2024-09-16T16:30:32.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:30:32.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182417 2024-09-16T16:30:32.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:30:32.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:30:32.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:30:32.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182417 2024-09-16T16:30:32.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182417 2024-09-16T16:30:32.596 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182417 2024-09-16T16:30:32.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2024-09-16T16:30:32.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:30:33.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182417 -lt 107374182417 2024-09-16T16:30:33.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:30:33.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018888 2024-09-16T16:30:33.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:30:33.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T16:30:33.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018888 2024-09-16T16:30:33.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:30:33.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018888 2024-09-16T16:30:33.059 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018888 2024-09-16T16:30:33.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2024-09-16T16:30:33.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T16:30:33.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018889 -lt 128849018888 2024-09-16T16:30:33.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:30:33.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:30:33.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:30:34.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:30:34.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:30:34.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:30:34.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:30:34.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:30:34.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:30:34.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:30:34.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:30:34.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:30:34.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:30:34.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:30:34.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:30:35.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:30:35.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:30:35.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:30:35.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-09-16T16:30:35.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:35.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-09-16T16:30:35.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:35.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-09-16T16:30:35.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:35.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-09-16T16:30:35.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:35.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-09-16T16:30:35.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:35.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-09-16T16:30:36.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:36.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-09-16T16:30:36.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:36.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-09-16T16:30:36.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:36.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-09-16T16:30:36.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:36.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-09-16T16:30:36.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:36.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-09-16T16:30:37.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:37.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-09-16T16:30:37.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:37.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-09-16T16:30:37.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:37.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-09-16T16:30:37.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:37.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-09-16T16:30:37.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:37.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-09-16T16:30:38.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:38.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-09-16T16:30:38.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:38.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-09-16T16:30:38.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:38.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-09-16T16:30:38.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:38.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-09-16T16:30:38.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:38.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-09-16T16:30:39.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:39.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-09-16T16:30:39.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:39.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-09-16T16:30:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:39.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-09-16T16:30:39.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:39.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-09-16T16:30:39.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:39.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-09-16T16:30:40.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:40.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-09-16T16:30:40.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:40.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-09-16T16:30:40.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:40.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-09-16T16:30:40.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:40.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-09-16T16:30:40.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:40.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-09-16T16:30:41.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:41.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-09-16T16:30:41.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:41.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-09-16T16:30:41.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:41.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-09-16T16:30:41.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:41.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-09-16T16:30:41.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:41.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-09-16T16:30:42.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:42.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-09-16T16:30:42.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:42.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-09-16T16:30:42.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:42.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-09-16T16:30:42.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:42.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-09-16T16:30:42.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:42.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-09-16T16:30:43.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:43.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-09-16T16:30:43.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:43.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-09-16T16:30:43.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:43.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-09-16T16:30:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-09-16T16:30:43.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:43.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-09-16T16:30:44.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:44.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-09-16T16:30:44.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:44.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-09-16T16:30:44.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:44.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-09-16T16:30:44.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:44.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-09-16T16:30:45.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:45.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-09-16T16:30:45.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:45.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-09-16T16:30:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:45.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-09-16T16:30:45.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:45.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-09-16T16:30:45.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:45.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-09-16T16:30:46.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:46.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-09-16T16:30:46.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:46.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-09-16T16:30:46.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:46.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-09-16T16:30:46.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:46.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-09-16T16:30:46.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:46.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-09-16T16:30:47.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:47.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-09-16T16:30:47.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:47.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-09-16T16:30:47.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:47.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-09-16T16:30:47.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:47.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-09-16T16:30:47.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:47.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-09-16T16:30:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:48.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-09-16T16:30:48.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:48.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-09-16T16:30:48.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:48.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-09-16T16:30:48.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:48.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-09-16T16:30:48.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:48.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-09-16T16:30:49.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:49.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-09-16T16:30:49.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:49.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-09-16T16:30:49.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:49.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-09-16T16:30:49.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:49.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-09-16T16:30:49.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:49.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-09-16T16:30:50.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:50.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-09-16T16:30:50.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:50.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-09-16T16:30:50.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:50.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-09-16T16:30:50.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:50.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-09-16T16:30:50.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:50.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-09-16T16:30:51.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:51.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-09-16T16:30:51.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:51.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-09-16T16:30:51.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:51.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-09-16T16:30:51.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:51.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-09-16T16:30:51.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:51.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-09-16T16:30:52.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:52.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-09-16T16:30:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:52.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-09-16T16:30:52.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:52.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-09-16T16:30:52.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:52.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-09-16T16:30:52.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:52.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-09-16T16:30:53.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:53.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-09-16T16:30:53.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:53.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-09-16T16:30:53.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:53.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-09-16T16:30:53.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:53.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-09-16T16:30:53.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:53.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-09-16T16:30:53.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:53.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-09-16T16:30:54.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:54.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-09-16T16:30:54.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:54.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-09-16T16:30:54.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:54.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-09-16T16:30:54.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:54.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-09-16T16:30:55.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:55.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-09-16T16:30:55.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:55.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-09-16T16:30:55.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:55.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-09-16T16:30:55.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:55.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-09-16T16:30:55.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:55.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-09-16T16:30:56.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:56.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-09-16T16:30:56.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:56.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-09-16T16:30:56.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:56.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-09-16T16:30:56.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:56.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-09-16T16:30:56.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:56.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-09-16T16:30:57.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:57.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-09-16T16:30:57.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:57.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-09-16T16:30:57.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:57.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-09-16T16:30:57.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:57.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-09-16T16:30:57.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:57.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-09-16T16:30:58.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:58.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-09-16T16:30:58.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:58.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-09-16T16:30:58.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:58.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-09-16T16:30:58.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:58.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-09-16T16:30:58.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:58.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-09-16T16:30:59.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:59.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-09-16T16:30:59.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:59.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-09-16T16:30:59.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:59.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-09-16T16:30:59.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:59.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-09-16T16:30:59.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:59.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-09-16T16:30:59.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:30:59.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-09-16T16:31:00.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:00.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-09-16T16:31:00.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:00.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-09-16T16:31:00.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:00.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-09-16T16:31:00.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:00.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-09-16T16:31:00.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:00.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-09-16T16:31:01.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:01.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-09-16T16:31:01.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:01.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-09-16T16:31:01.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:01.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-09-16T16:31:01.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:01.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-09-16T16:31:01.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:01.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-09-16T16:31:02.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:02.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-09-16T16:31:02.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:02.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-09-16T16:31:02.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:02.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-09-16T16:31:02.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:02.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-09-16T16:31:02.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:02.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-09-16T16:31:03.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:03.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-09-16T16:31:03.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:03.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-09-16T16:31:03.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:03.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-09-16T16:31:03.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:03.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-09-16T16:31:03.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:03.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-09-16T16:31:04.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:04.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-09-16T16:31:04.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:04.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-09-16T16:31:04.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:04.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-09-16T16:31:04.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:04.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-09-16T16:31:04.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:04.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-09-16T16:31:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-09-16T16:31:05.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:05.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-09-16T16:31:05.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:05.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-09-16T16:31:05.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:05.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-09-16T16:31:06.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:06.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-09-16T16:31:06.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:06.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-09-16T16:31:06.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:06.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-09-16T16:31:06.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:06.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-09-16T16:31:06.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:06.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-09-16T16:31:07.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:07.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-09-16T16:31:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-09-16T16:31:07.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:07.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-09-16T16:31:07.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:07.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-09-16T16:31:08.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:08.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-09-16T16:31:08.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:08.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-09-16T16:31:08.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:08.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-09-16T16:31:08.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:08.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-09-16T16:31:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:08.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-09-16T16:31:09.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:09.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-09-16T16:31:09.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:09.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-09-16T16:31:09.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:09.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-09-16T16:31:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-09-16T16:31:09.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:09.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-09-16T16:31:10.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:10.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-09-16T16:31:10.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:10.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-09-16T16:31:10.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:10.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-09-16T16:31:10.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:10.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-09-16T16:31:10.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:10.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-09-16T16:31:11.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:11.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-09-16T16:31:11.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:11.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-09-16T16:31:11.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:11.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-09-16T16:31:11.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:11.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-09-16T16:31:11.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:11.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-09-16T16:31:12.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:12.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-09-16T16:31:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:12.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-09-16T16:31:12.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:12.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-09-16T16:31:12.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:12.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-09-16T16:31:12.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:12.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-09-16T16:31:13.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:13.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-09-16T16:31:13.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:13.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-09-16T16:31:13.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:13.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-09-16T16:31:13.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:13.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-09-16T16:31:13.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:13.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-09-16T16:31:14.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:14.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-09-16T16:31:14.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:14.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-09-16T16:31:14.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:14.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-09-16T16:31:14.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:14.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-09-16T16:31:14.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:14.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-09-16T16:31:15.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:15.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-09-16T16:31:15.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:31:15.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-09-16T16:31:15.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-09-16T16:31:15.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:31:15.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:31:15.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:31:15.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:31:15.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-09-16T16:31:15.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-09-16T16:31:15.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-16T16:31:15.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-16T16:31:15.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T16:31:15.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-16T16:31:16.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-09-16T16:31:16.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-09-16T16:31:16.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-16T16:31:16.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-16T16:31:16.282 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-16T16:31:16.282 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:31:16.282 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:31:16.282 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:31:16.282 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:31:16.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-16T16:31:16.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-16T16:31:16.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-16T16:31:17.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-09-16T16:31:17.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-09-16T16:31:17.936 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is set 2024-09-16T16:31:17.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-09-16T16:31:17.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 276022 2024-09-16T16:31:17.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-09-16T16:31:18.421 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T16:31:18.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-09-16T16:31:19.455 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T16:31:19.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-09-16T16:31:20.511 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is unset 2024-09-16T16:31:20.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-09-16T16:31:20.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:31:20.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:31:20.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:31:20.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:31:21.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-16T16:31:21.469 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T16:31:21.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-09-16T16:31:23.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-09-16T16:31:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:31:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:31:23.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:31:23.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:31:23.488 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:31:23.488 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:31:23.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:31:23.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:31:23.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:31:23.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:31:23.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:31:23.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:31:23.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:31:23.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:31:23.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:31:24.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:31:24.045 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:31:24.045 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:31:24.045 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:31:24.045 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T16:31:24.045 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T16:31:24.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:31:24.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:31:24.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:31:24.168 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-16T16:31:24.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T16:31:24.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T16:31:24.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T16:31:24.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:31:24.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:31:24.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673011 2024-09-16T16:31:24.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673011 2024-09-16T16:31:24.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949673011' 2024-09-16T16:31:24.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:31:24.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:31:24.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509482 2024-09-16T16:31:24.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509482 2024-09-16T16:31:24.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949673011 2-64424509482' 2024-09-16T16:31:24.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:31:24.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:31:24.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345955 2024-09-16T16:31:24.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345955 2024-09-16T16:31:24.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949673011 2-64424509482 3-85899345955' 2024-09-16T16:31:24.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:31:24.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:31:24.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182430 2024-09-16T16:31:24.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182430 2024-09-16T16:31:24.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949673011 2-64424509482 3-85899345955 4-107374182430' 2024-09-16T16:31:24.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:31:24.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T16:31:25.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018901 2024-09-16T16:31:25.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018901 2024-09-16T16:31:25.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949673011 2-64424509482 3-85899345955 4-107374182430 5-128849018901' 2024-09-16T16:31:25.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:31:25.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673011 2024-09-16T16:31:25.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:31:25.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:31:25.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673011 2024-09-16T16:31:25.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:31:25.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673011 2024-09-16T16:31:25.059 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673011 2024-09-16T16:31:25.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2024-09-16T16:31:25.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:31:25.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673011 -lt 42949673011 2024-09-16T16:31:25.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:31:25.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509482 2024-09-16T16:31:25.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:31:25.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:31:25.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509482 2024-09-16T16:31:25.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:31:25.493 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509482 2024-09-16T16:31:25.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509482 2024-09-16T16:31:25.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-09-16T16:31:25.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:31:25.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509480 -lt 64424509482 2024-09-16T16:31:25.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:31:26.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:31:26.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:31:27.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509482 -lt 64424509482 2024-09-16T16:31:27.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:31:27.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:31:27.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345955 2024-09-16T16:31:27.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:31:27.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345955 2024-09-16T16:31:27.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:31:27.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345955 2024-09-16T16:31:27.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345955' 2024-09-16T16:31:27.387 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345955 2024-09-16T16:31:27.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:31:27.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345955 -lt 85899345955 2024-09-16T16:31:27.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:31:27.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182430 2024-09-16T16:31:27.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:31:27.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:31:27.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182430 2024-09-16T16:31:27.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:31:27.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182430 2024-09-16T16:31:27.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182430' 2024-09-16T16:31:27.875 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182430 2024-09-16T16:31:27.875 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:31:28.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182430 -lt 107374182430 2024-09-16T16:31:28.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:31:28.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:31:28.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018901 2024-09-16T16:31:28.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T16:31:28.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018901 2024-09-16T16:31:28.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:31:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018901 2024-09-16T16:31:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018901' 2024-09-16T16:31:28.308 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018901 2024-09-16T16:31:28.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T16:31:28.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018902 -lt 128849018901 2024-09-16T16:31:28.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:31:28.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:31:28.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:31:29.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:31:29.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:31:29.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:31:29.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:31:29.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:31:29.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:31:29.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:31:29.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:31:29.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:31:29.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:31:29.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:31:29.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:31:30.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:31:30.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T16:31:30.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T16:31:30.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T16:31:30.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:31:30.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:31:30.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:31:30.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:31:30.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:31:30.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:31:30.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:31:30.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:31:30.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:31:30.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:31:31.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:31:31.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:31:31.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:31:31.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:31:31.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:31:31.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:31:31.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:31:31.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:31:31.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:31:31.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:31:31.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:31:31.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:31:31.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:31:32.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15 2024-09-16T16:31:32.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15 '!=' null 2024-09-16T16:31:32.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:31:32.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:31:32.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:31:32.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:31:32.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:31:32.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:31:32.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:31:32.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:31:32.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:31:32.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:31:32.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:31:32.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:31:32.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:31:32.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:31:32.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:31:33.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:31:33.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:31:33.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:31:33.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:31:33.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:31:33.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:31:33.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:31:33.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:31:33.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:31:34.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12 2024-09-16T16:31:34.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12 '!=' null 2024-09-16T16:31:34.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:31:34.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:31:34.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:31:34.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:31:34.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:31:34.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:31:34.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:31:34.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:31:34.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:31:34.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:31:34.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:31:34.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:31:34.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:31:34.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:31:34.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:31:35.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:31:35.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:31:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:31:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:31:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:31:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:31:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:31:35.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:31:35.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:31:35.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-16T16:31:35.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-16T16:31:35.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:31:35.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:31:35.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:31:35.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:31:35.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:31:35.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:31:35.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:31:35.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:31:35.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:31:35.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:31:35.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:31:36.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:31:36.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:31:36.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:31:36.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:31:36.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:31:36.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:31:36.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:31:36.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:31:36.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:31:36.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:31:36.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:31:36.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:31:36.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:31:37.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15 2024-09-16T16:31:37.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15 '!=' null 2024-09-16T16:31:37.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:31:37.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:31:37.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:31:37.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:31:37.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:31:37.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:31:37.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:31:37.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:31:37.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:31:37.538 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:31:37.538 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:31:37.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:31:37.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:31:37.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:31:37.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:31:38.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 erasure 200 0 0 0 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-16T16:31:38.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-16T16:31:38.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-16T16:31:38.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-16T16:31:38.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-09-16T16:31:38.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-09-16T16:31:38.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-16T16:31:38.485 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:31:38.485 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-16T16:31:38.486 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-16T16:31:38.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,2]/[1,NONE,2]' 2024-09-16T16:31:38.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-16T16:31:38.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:31:38.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:31:38.505 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-09-16T16:31:38.506 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-16T16:31:38.506 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-16T16:31:38.506 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:31:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-09-16T16:31:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-09-16T16:31:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-16T16:31:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:31:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-16T16:31:38.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:31:38.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:31:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:31:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-16T16:31:38.517 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:31:38.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-16T16:31:38.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-16T16:31:38.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:31:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:31:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:31:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:31:38.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-16T16:31:38.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:31:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:31:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-09-16T16:31:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-16T16:31:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-16T16:31:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:31:38.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-09-16T16:31:38.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-09-16T16:31:38.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-16T16:31:38.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:31:38.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-16T16:31:38.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:31:38.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:31:38.544 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:31:38.544 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-16T16:31:38.544 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:31:38.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-16T16:31:38.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-16T16:31:38.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:31:38.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:31:38.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:31:38.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:31:38.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-16T16:31:38.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-09-16T16:31:38.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T16:31:38.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T16:31:39.112 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T16:31:39.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-09-16T16:31:39.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:31:39.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:31:39.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:31:39.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:31:39.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:31:44.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:31:44.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-16T16:31:44.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:31:44.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:31:44.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:31:44.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:31:44.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:31:44.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:31:44.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:31:44.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:31:44.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:31:44.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:31:44.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:31:44.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:31:44.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:31:44.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:31:44.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:31:44.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:31:44.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:31:44.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:31:44.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:31:44.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:31:44.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:31:44.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:31:44.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:31:44.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:31:44.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:31:44.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:31:44.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:31:44.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:31:44.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-16T16:31:44.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-16T16:31:44.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-16T16:31:44.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-16T16:31:44.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:31:44.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:31:44.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:31:44.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:31:44.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:31:44.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:31:44.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:31:44.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:31:44.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:31:44.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:31:44.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:31:44.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:31:44.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:31:44.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:31:44.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:31:44.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:31:44.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:31:44.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:31:44.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:31:44.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:31:44.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:31:44.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:31:44.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:31:44.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:31:44.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:31:44.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:31:44.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:31:44.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:31:44.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-16T16:31:44.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:31:44.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:31:44.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:31:44.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274941 2024-09-16T16:31:44.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:31:44.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T16:31:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:31:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-16T16:31:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_remapped td/osd-recovery-stats 2024-09-16T16:31:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:384: TEST_recovery_erasure_remapped: local dir=td/osd-recovery-stats 2024-09-16T16:31:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:386: TEST_recovery_erasure_remapped: run_mon td/osd-recovery-stats a 2024-09-16T16:31:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-16T16:31:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:31:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:31:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:31:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-16T16:31:44.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-16T16:31:44.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:31:44.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:31:44.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:31:44.914 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:31:44.914 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:31:44.914 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:31:44.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:31:44.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:31:45.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:31:45.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:31:45.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:31:45.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:31:45.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:31:45.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:31:45.136 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:31:45.137 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:31:45.137 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:31:45.137 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:31:45.137 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:31:45.137 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:31:45.138 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:31:45.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:31:45.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get fsid 2024-09-16T16:31:45.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:31:45.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:31:45.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:31:45.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:31:45.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:31:45.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:31:45.259 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:31:45.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:31:45.259 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:31:45.259 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:31:45.259 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:31:45.259 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:31:45.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:31:45.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get mon_host 2024-09-16T16:31:45.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:387: TEST_recovery_erasure_remapped: run_mgr td/osd-recovery-stats x 2024-09-16T16:31:45.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-16T16:31:45.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:31:45.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:31:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:31:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-16T16:31:45.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:31:46.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:31:46.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:31:46.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:31:46.291 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:31:46.291 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:31:46.291 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:31:46.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:31:46.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:31:46.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:31:46.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:388: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 0 2024-09-16T16:31:46.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:31:46.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:31:46.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:31:46.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:31:46.330 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:31:46.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:31:46.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:31:46.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-16T16:31:46.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:31:46.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8bb4958-bcf2-48b0-9d37-07d9e54f1e2b 2024-09-16T16:31:46.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c8bb4958-bcf2-48b0-9d37-07d9e54f1e2b' 2024-09-16T16:31:46.333 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 c8bb4958-bcf2-48b0-9d37-07d9e54f1e2b 2024-09-16T16:31:46.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:31:46.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQByXehmtf3ZFBAA9FyOEQ6vK2pmEfTo7cnaRw== 2024-09-16T16:31:46.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQByXehmtf3ZFBAA9FyOEQ6vK2pmEfTo7cnaRw=="}' 2024-09-16T16:31:46.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8bb4958-bcf2-48b0-9d37-07d9e54f1e2b -i td/osd-recovery-stats/0/new.json 2024-09-16T16:31:46.714 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:31:46.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-16T16:31:46.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQByXehmtf3ZFBAA9FyOEQ6vK2pmEfTo7cnaRw== --osd-uuid c8bb4958-bcf2-48b0-9d37-07d9e54f1e2b 2024-09-16T16:31:46.767 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:46.765+0000 7f86c952e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:31:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:46.769+0000 7f86c952e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:31:46.771 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:46.769+0000 7f86c952e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:31:46.771 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:46.769+0000 7f86c952e7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:31:46.772 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:46.769+0000 7f86c952e7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:31:46.772 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:46.769+0000 7f86c952e7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:31:46.854 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:46.853+0000 7f86c952e7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-16T16:31:53.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-16T16:31:53.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:31:53.033 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:31:53.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:31:53.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:31:53.589 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:31:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:31:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:31:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:31:53.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:31:53.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:31:53.624 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:53.621+0000 7fce825d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:31:53.626 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:53.625+0000 7fce825d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:31:53.627 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:53.625+0000 7fce825d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:31:54.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:31:54.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:31:54.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:31:54.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:31:54.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:31:54.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:31:54.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:31:54.035 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:31:54.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:31:54.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:31:54.207 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:54.205+0000 7fce825d37c0 -1 Falling back to public interface 2024-09-16T16:31:54.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:31:55.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:31:55.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:31:55.527 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:31:55.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:31:55.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:31:55.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:31:55.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:31:56.808 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:31:56.805+0000 7fce825d37c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:31:56.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:31:56.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:31:56.960 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:31:56.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:31:56.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:31:56.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:31:57.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:31:58.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:31:58.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:31:58.384 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:31:58.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:31:58.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:31:58.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:31:58.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:31:59.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:31:59.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:31:59.827 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:31:59.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:31:59.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:31:59.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:00.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:01.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:01.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:01.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:32:01.258 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:32:01.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:01.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:01.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:02.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:02.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:02.794 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:32:02.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:32:02.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:02.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:03.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:04.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:04.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:04.258 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:32:04.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:32:04.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:04.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:04.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:05.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:05.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:05.694 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:32:05.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:32:05.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:05.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:06.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:07.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:07.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:07.151 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:32:07.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:32:07.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:07.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:07.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:08.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:08.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:08.692 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:32:08.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:32:08.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:08.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:09.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:10.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:10.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:10.138 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:32:10.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:32:10.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:10.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:10.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:11.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:11.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:11.681 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:32:11.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:32:11.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:11.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:13.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:13.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:13.393 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:32:13.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:32:13.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:13.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:14.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:15.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:15.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:15.106 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:32:15.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:32:15.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:15.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:15.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:16.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:16.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:16.544 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:32:16.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:32:16.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:16.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:17.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:18.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:18.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:18.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:32:18.394 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:32:18.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:18.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:19.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:20.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:20.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:20.246 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:32:20.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:32:20.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:20.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:32:20.690 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1636535988,v1:127.0.0.1:6803/1636535988] [v2:127.0.0.1:6804/1636535988,v1:127.0.0.1:6805/1636535988] exists,up c8bb4958-bcf2-48b0-9d37-07d9e54f1e2b 2024-09-16T16:32:20.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:32:20.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:32:20.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:389: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 1 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-16T16:32:20.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:32:20.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:32:20.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:32:20.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:32:20.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:32:20.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:32:20.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:32:20.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:32:20.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:32:20.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:32:20.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:32:20.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-16T16:32:20.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:32:20.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=069ef2db-558a-4b26-89d2-ec6047054957 2024-09-16T16:32:20.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 069ef2db-558a-4b26-89d2-ec6047054957' 2024-09-16T16:32:20.697 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 069ef2db-558a-4b26-89d2-ec6047054957 2024-09-16T16:32:20.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:32:20.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUXehmxObVKhAABc3rtZXpTnGzIFmYSeg7Mg== 2024-09-16T16:32:20.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUXehmxObVKhAABc3rtZXpTnGzIFmYSeg7Mg=="}' 2024-09-16T16:32:20.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 069ef2db-558a-4b26-89d2-ec6047054957 -i td/osd-recovery-stats/1/new.json 2024-09-16T16:32:21.268 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:32:21.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-16T16:32:21.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUXehmxObVKhAABc3rtZXpTnGzIFmYSeg7Mg== --osd-uuid 069ef2db-558a-4b26-89d2-ec6047054957 2024-09-16T16:32:21.320 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:21.317+0000 7fceea56e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:32:21.335 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:21.333+0000 7fceea56e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:32:21.337 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:21.333+0000 7fceea56e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:32:21.337 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:21.333+0000 7fceea56e7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:32:21.337 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:21.333+0000 7fceea56e7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:32:21.337 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:21.333+0000 7fceea56e7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:32:21.427 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:21.425+0000 7fceea56e7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-16T16:32:27.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-16T16:32:27.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:32:27.208 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:32:27.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:32:27.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:32:27.773 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:32:27.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:32:27.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:32:27.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:32:27.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:32:27.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:32:27.799 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:27.797+0000 7f4b126737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:32:27.842 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:27.841+0000 7f4b126737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:32:27.843 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:27.841+0000 7f4b126737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:32:28.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:32:28.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:32:28.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:32:28.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:32:28.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:32:28.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:28.209 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:32:28.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:32:28.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:28.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:28.395 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:28.393+0000 7f4b126737c0 -1 Falling back to public interface 2024-09-16T16:32:28.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:29.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:29.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:29.623 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:32:29.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:32:29.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:29.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:30.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:30.960 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:30.957+0000 7f4b126737c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:32:31.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:31.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:31.094 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:32:31.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:32:31.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:31.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:31.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:32.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:32.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:32.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:32:32.767 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:32:32.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:32.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:33.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:34.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:34.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:34.191 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:32:34.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:32:34.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:34.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:34.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:35.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:35.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:35.623 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:32:35.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:32:35.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:35.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:36.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:37.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:37.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:37.091 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:32:37.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:37.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:37.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:38.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:38.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:38.663 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:32:38.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:32:38.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:38.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:39.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:40.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:40.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:40.210 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:32:40.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:32:40.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:40.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:40.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:41.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:41.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:41.651 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:32:41.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:32:41.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:41.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:42.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:43.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:43.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:43.155 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:32:43.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:32:43.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:43.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:44.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:45.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:45.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:45.145 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:32:45.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:32:45.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:45.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:46.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:47.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:47.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:47.228 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:32:47.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:32:47.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:47.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:48.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:49.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:49.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:49.127 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:32:49.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:32:49.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:49.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:50.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:51.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:51.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:51.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:32:51.172 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:32:51.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:51.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:51.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:52.638 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:32:52.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:32:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:52.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:53.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:32:54.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:32:54.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:32:54.377 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:32:54.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:32:54.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:32:54.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:32:54.887 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2925466688,v1:127.0.0.1:6811/2925466688] [v2:127.0.0.1:6812/2925466688,v1:127.0.0.1:6813/2925466688] exists,up 069ef2db-558a-4b26-89d2-ec6047054957 2024-09-16T16:32:54.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:32:54.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:32:54.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:32:54.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:390: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 2 2024-09-16T16:32:54.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:32:54.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:32:54.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:32:54.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:32:54.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:32:54.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:32:54.891 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:32:54.891 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:32:54.891 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:32:54.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:32:54.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:32:54.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-16T16:32:54.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:32:54.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0b197bd0-b717-452c-8644-804eb03aec1d 2024-09-16T16:32:54.894 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 0b197bd0-b717-452c-8644-804eb03aec1d 2024-09-16T16:32:54.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0b197bd0-b717-452c-8644-804eb03aec1d' 2024-09-16T16:32:54.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:32:54.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2XehmjnBONhAA/pY/Taw+6ne8FXFaudrNYw== 2024-09-16T16:32:54.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2XehmjnBONhAA/pY/Taw+6ne8FXFaudrNYw=="}' 2024-09-16T16:32:54.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0b197bd0-b717-452c-8644-804eb03aec1d -i td/osd-recovery-stats/2/new.json 2024-09-16T16:32:55.547 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:32:55.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-16T16:32:55.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2XehmjnBONhAA/pY/Taw+6ne8FXFaudrNYw== --osd-uuid 0b197bd0-b717-452c-8644-804eb03aec1d 2024-09-16T16:32:55.600 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:55.596+0000 7fb321dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:32:55.602 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:55.600+0000 7fb321dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:32:55.603 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:55.600+0000 7fb321dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:32:55.604 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:55.600+0000 7fb321dae7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:32:55.604 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:55.600+0000 7fb321dae7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:32:55.604 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:55.600+0000 7fb321dae7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:32:55.635 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:32:55.632+0000 7fb321dae7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-16T16:33:01.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-16T16:33:01.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:33:01.425 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T16:33:01.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T16:33:01.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:33:01.986 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T16:33:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T16:33:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:33:01.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:33:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:33:01.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:33:02.021 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:02.016+0000 7f40f6c727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:33:02.050 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:02.048+0000 7f40f6c727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:33:02.052 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:02.048+0000 7f40f6c727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:33:02.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T16:33:02.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:33:02.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T16:33:02.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:33:02.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:33:02.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:02.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:33:02.413 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:33:02.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:02.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:02.619 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:02.616+0000 7f40f6c727c0 -1 Falling back to public interface 2024-09-16T16:33:02.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:03.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:03.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:03.900 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:33:03.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:33:03.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:03.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:04.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:05.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:05.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:05.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:33:05.319 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:33:05.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:05.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:05.336 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:05.332+0000 7f40f6c727c0 -1 osd.2 0 log_to_monitors true 2024-09-16T16:33:05.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:06.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:06.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:06.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:33:06.861 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:33:06.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:06.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:07.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:08.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:08.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:08.364 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:33:08.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:33:08.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:08.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:08.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:09.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:09.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:09.841 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:33:09.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:33:09.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:09.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:10.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:11.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:11.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:11.386 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:33:11.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:33:11.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:11.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:11.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:12.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:12.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:12.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:33:12.830 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:33:12.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:12.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:13.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:14.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:14.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:14.337 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:33:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:33:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:14.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:14.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:15.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:15.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:15.785 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:33:15.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:33:15.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:15.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:16.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:17.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:17.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:17.276 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:33:17.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:33:17.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:17.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:17.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:18.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:18.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:18.808 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:33:18.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:33:18.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:18.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:20.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:21.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:21.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:21.154 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:33:21.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:33:21.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:21.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:21.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:22.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:22.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:22.655 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:33:22.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:33:22.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:22.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:23.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:24.357 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:33:24.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:33:24.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:24.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:25.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:26.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:26.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:26.345 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:33:26.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:33:26.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:26.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:27.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:28.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:28.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:28.079 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:33:28.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:33:28.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:28.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:28.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:29.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:29.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:29.638 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:33:29.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:33:29.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:29.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:33:30.123 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/627636538,v1:127.0.0.1:6819/627636538] [v2:127.0.0.1:6820/627636538,v1:127.0.0.1:6821/627636538] exists,up 0b197bd0-b717-452c-8644-804eb03aec1d 2024-09-16T16:33:30.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:33:30.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:33:30.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:391: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 3 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-16T16:33:30.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:33:30.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:33:30.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:33:30.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:33:30.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:33:30.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:33:30.129 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:33:30.129 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:33:30.129 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:33:30.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:33:30.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:33:30.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:33:30.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:33:30.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:33:30.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:33:30.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:33:30.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:33:30.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:33:30.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:33:30.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:33:30.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:33:30.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:33:30.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-16T16:33:30.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:33:30.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1454c5d9-622a-40d9-8b50-f327720f7a47 2024-09-16T16:33:30.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 1454c5d9-622a-40d9-8b50-f327720f7a47' 2024-09-16T16:33:30.131 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 1454c5d9-622a-40d9-8b50-f327720f7a47 2024-09-16T16:33:30.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:33:30.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDaXehmUo83CRAAUC4Vam7bKg4X28cN1F8MBQ== 2024-09-16T16:33:30.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDaXehmUo83CRAAUC4Vam7bKg4X28cN1F8MBQ=="}' 2024-09-16T16:33:30.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1454c5d9-622a-40d9-8b50-f327720f7a47 -i td/osd-recovery-stats/3/new.json 2024-09-16T16:33:30.691 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:33:30.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-16T16:33:30.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaXehmUo83CRAAUC4Vam7bKg4X28cN1F8MBQ== --osd-uuid 1454c5d9-622a-40d9-8b50-f327720f7a47 2024-09-16T16:33:30.747 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:30.743+0000 7fda96b2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:33:30.749 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:30.743+0000 7fda96b2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:33:30.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:30.747+0000 7fda96b2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:33:30.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:30.747+0000 7fda96b2f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:33:30.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:30.747+0000 7fda96b2f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:33:30.751 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:30.747+0000 7fda96b2f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:33:30.829 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:30.823+0000 7fda96b2f7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-16T16:33:36.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-16T16:33:36.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:33:36.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T16:33:36.693 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T16:33:36.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:33:37.261 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T16:33:37.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T16:33:37.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:33:37.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:33:37.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:33:37.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:33:37.294 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:37.291+0000 7ff21f5b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:33:37.320 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:37.315+0000 7ff21f5b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:33:37.321 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:37.319+0000 7ff21f5b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:33:37.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T16:33:37.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:33:37.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T16:33:37.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:33:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:33:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:37.686 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:33:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:33:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:37.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:37.907 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:37.903+0000 7ff21f5b37c0 -1 Falling back to public interface 2024-09-16T16:33:38.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:39.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:39.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:39.145 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:33:39.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:33:39.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:39.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:39.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:40.507 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:33:40.503+0000 7ff21f5b37c0 -1 osd.3 0 log_to_monitors true 2024-09-16T16:33:40.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:40.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:40.612 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:33:40.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:33:40.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:40.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:41.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:42.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:42.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:42.275 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:33:42.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:33:42.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:42.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:42.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:43.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:43.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:43.718 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:33:43.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:33:43.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:43.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:44.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:45.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:45.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:45.358 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:33:45.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:33:45.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:45.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:45.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:46.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:46.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:46.833 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:33:46.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:33:46.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:46.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:47.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:48.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:48.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:48.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:33:48.471 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:33:48.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:48.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:48.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:49.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:49.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:49.907 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:33:49.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:33:49.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:49.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:50.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:51.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:51.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:51.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:33:51.392 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:33:51.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:51.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:52.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:52.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:52.928 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:33:52.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:33:52.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:52.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:53.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:54.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:54.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:54.766 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:33:54.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:33:54.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:54.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:55.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:56.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:56.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:56.518 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:33:56.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:33:56.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:56.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:57.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:33:58.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:33:58.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:33:58.393 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:33:58.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:33:58.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:33:58.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:33:59.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:34:00.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:34:00.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:34:00.110 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:34:00.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:34:00.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:34:00.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:34:00.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:34:01.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:34:01.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:34:01.801 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:34:01.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:34:01.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:34:01.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:34:02.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:34:03.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:34:03.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:34:03.546 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:34:03.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:34:03.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:34:03.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:34:04.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:34:05.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:34:05.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:34:05.297 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:34:05.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:34:05.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:34:05.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:34:05.820 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3364758376,v1:127.0.0.1:6827/3364758376] [v2:127.0.0.1:6828/3364758376,v1:127.0.0.1:6829/3364758376] exists,up 1454c5d9-622a-40d9-8b50-f327720f7a47 2024-09-16T16:34:05.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:34:05.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:34:05.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:34:05.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:393: TEST_recovery_erasure_remapped: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T16:34:06.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:394: TEST_recovery_erasure_remapped: create_pool test 1 1 erasure myprofile 2024-09-16T16:34:06.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-16T16:34:07.314 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T16:34:07.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:34:08.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:395: TEST_recovery_erasure_remapped: ceph osd pool set test min_size 2 2024-09-16T16:34:09.301 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 2 2024-09-16T16:34:09.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:397: TEST_recovery_erasure_remapped: wait_for_clean 2024-09-16T16:34:09.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:34:09.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:34:09.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:34:09.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:34:09.328 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:34:09.328 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:34:09.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:34:09.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:34:09.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:34:09.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:34:09.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:34:09.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:34:09.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:34:09.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:34:09.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:34:09.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:34:09.947 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:34:09.947 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:34:09.947 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T16:34:09.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:34:09.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:34:09.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:34:10.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836509 2024-09-16T16:34:10.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836509 2024-09-16T16:34:10.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836509' 2024-09-16T16:34:10.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:34:10.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:34:10.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672981 2024-09-16T16:34:10.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672981 2024-09-16T16:34:10.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836509 1-42949672981' 2024-09-16T16:34:10.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:34:10.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:34:10.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509458 2024-09-16T16:34:10.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509458 2024-09-16T16:34:10.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509458' 2024-09-16T16:34:10.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:34:10.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:34:10.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345927 2024-09-16T16:34:10.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345927 2024-09-16T16:34:10.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509458 3-85899345927' 2024-09-16T16:34:10.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:34:10.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836509 2024-09-16T16:34:10.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:34:10.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:34:10.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836509 2024-09-16T16:34:10.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:34:10.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836509 2024-09-16T16:34:10.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-16T16:34:10.608 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836509 2024-09-16T16:34:10.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:34:11.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836507 -lt 21474836509 2024-09-16T16:34:11.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:34:12.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:34:12.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:34:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-16T16:34:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:34:12.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672981 2024-09-16T16:34:12.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:34:12.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:34:12.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672981 2024-09-16T16:34:12.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:34:12.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672981 2024-09-16T16:34:12.441 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672981 2024-09-16T16:34:12.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-16T16:34:12.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:34:12.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672982 -lt 42949672981 2024-09-16T16:34:12.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:34:12.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509458 2024-09-16T16:34:12.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:34:12.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:34:12.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509458 2024-09-16T16:34:12.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:34:12.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509458 2024-09-16T16:34:12.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-16T16:34:12.851 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509458 2024-09-16T16:34:12.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:34:13.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-16T16:34:13.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:34:13.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345927 2024-09-16T16:34:13.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:34:13.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:34:13.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:34:13.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345927 2024-09-16T16:34:13.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345927 2024-09-16T16:34:13.269 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345927 2024-09-16T16:34:13.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2024-09-16T16:34:13.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:34:13.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345927 -lt 85899345927 2024-09-16T16:34:13.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:34:13.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:34:13.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:34:14.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:34:14.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:34:14.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:34:14.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:34:14.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:34:14.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:34:14.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:34:14.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:34:14.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:34:14.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:34:14.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:34:14.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:34:15.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:34:15.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:34:15.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:34:15.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: seq 1 200 2024-09-16T16:34:15.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:15.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj1 /dev/null 2024-09-16T16:34:15.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:15.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj2 /dev/null 2024-09-16T16:34:15.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:15.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj3 /dev/null 2024-09-16T16:34:15.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:15.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj4 /dev/null 2024-09-16T16:34:16.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:16.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj5 /dev/null 2024-09-16T16:34:16.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:16.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj6 /dev/null 2024-09-16T16:34:16.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:16.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj7 /dev/null 2024-09-16T16:34:16.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:16.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj8 /dev/null 2024-09-16T16:34:16.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:16.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj9 /dev/null 2024-09-16T16:34:16.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:16.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj10 /dev/null 2024-09-16T16:34:17.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:17.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj11 /dev/null 2024-09-16T16:34:17.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:17.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj12 /dev/null 2024-09-16T16:34:17.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:17.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj13 /dev/null 2024-09-16T16:34:17.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:17.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj14 /dev/null 2024-09-16T16:34:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:17.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj15 /dev/null 2024-09-16T16:34:18.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:18.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj16 /dev/null 2024-09-16T16:34:18.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:18.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj17 /dev/null 2024-09-16T16:34:18.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:18.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj18 /dev/null 2024-09-16T16:34:18.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:18.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj19 /dev/null 2024-09-16T16:34:18.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:18.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj20 /dev/null 2024-09-16T16:34:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:19.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj21 /dev/null 2024-09-16T16:34:19.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:19.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj22 /dev/null 2024-09-16T16:34:19.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:19.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj23 /dev/null 2024-09-16T16:34:19.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:19.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj24 /dev/null 2024-09-16T16:34:19.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:19.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj25 /dev/null 2024-09-16T16:34:20.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:20.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj26 /dev/null 2024-09-16T16:34:20.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:20.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj27 /dev/null 2024-09-16T16:34:20.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:20.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj28 /dev/null 2024-09-16T16:34:20.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:20.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj29 /dev/null 2024-09-16T16:34:20.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:20.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj30 /dev/null 2024-09-16T16:34:21.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:21.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj31 /dev/null 2024-09-16T16:34:21.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:21.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj32 /dev/null 2024-09-16T16:34:21.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:21.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj33 /dev/null 2024-09-16T16:34:21.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:21.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj34 /dev/null 2024-09-16T16:34:21.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:21.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj35 /dev/null 2024-09-16T16:34:22.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:22.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj36 /dev/null 2024-09-16T16:34:22.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:22.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj37 /dev/null 2024-09-16T16:34:22.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:22.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj38 /dev/null 2024-09-16T16:34:22.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:22.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj39 /dev/null 2024-09-16T16:34:22.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:22.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj40 /dev/null 2024-09-16T16:34:23.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:23.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj41 /dev/null 2024-09-16T16:34:23.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:23.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj42 /dev/null 2024-09-16T16:34:23.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:23.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj43 /dev/null 2024-09-16T16:34:23.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:23.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj44 /dev/null 2024-09-16T16:34:23.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:23.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj45 /dev/null 2024-09-16T16:34:24.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:24.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj46 /dev/null 2024-09-16T16:34:24.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:24.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj47 /dev/null 2024-09-16T16:34:24.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:24.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj48 /dev/null 2024-09-16T16:34:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj49 /dev/null 2024-09-16T16:34:24.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:24.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj50 /dev/null 2024-09-16T16:34:25.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:25.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj51 /dev/null 2024-09-16T16:34:25.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:25.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj52 /dev/null 2024-09-16T16:34:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj53 /dev/null 2024-09-16T16:34:25.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:25.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj54 /dev/null 2024-09-16T16:34:25.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:25.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj55 /dev/null 2024-09-16T16:34:25.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:25.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj56 /dev/null 2024-09-16T16:34:26.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:26.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj57 /dev/null 2024-09-16T16:34:26.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:26.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj58 /dev/null 2024-09-16T16:34:26.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:26.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj59 /dev/null 2024-09-16T16:34:26.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:26.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj60 /dev/null 2024-09-16T16:34:26.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:26.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj61 /dev/null 2024-09-16T16:34:27.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:27.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj62 /dev/null 2024-09-16T16:34:27.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:27.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj63 /dev/null 2024-09-16T16:34:27.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:27.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj64 /dev/null 2024-09-16T16:34:27.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:27.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj65 /dev/null 2024-09-16T16:34:27.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:27.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj66 /dev/null 2024-09-16T16:34:28.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:28.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj67 /dev/null 2024-09-16T16:34:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:28.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj68 /dev/null 2024-09-16T16:34:28.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:28.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj69 /dev/null 2024-09-16T16:34:28.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:28.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj70 /dev/null 2024-09-16T16:34:28.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:28.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj71 /dev/null 2024-09-16T16:34:29.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:29.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj72 /dev/null 2024-09-16T16:34:29.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:29.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj73 /dev/null 2024-09-16T16:34:29.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:29.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj74 /dev/null 2024-09-16T16:34:29.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:29.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj75 /dev/null 2024-09-16T16:34:29.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:29.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj76 /dev/null 2024-09-16T16:34:29.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:29.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj77 /dev/null 2024-09-16T16:34:30.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:30.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj78 /dev/null 2024-09-16T16:34:30.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:30.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj79 /dev/null 2024-09-16T16:34:30.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:30.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj80 /dev/null 2024-09-16T16:34:30.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:30.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj81 /dev/null 2024-09-16T16:34:30.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:30.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj82 /dev/null 2024-09-16T16:34:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj83 /dev/null 2024-09-16T16:34:31.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:31.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj84 /dev/null 2024-09-16T16:34:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj85 /dev/null 2024-09-16T16:34:31.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:31.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj86 /dev/null 2024-09-16T16:34:31.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:31.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj87 /dev/null 2024-09-16T16:34:32.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:32.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj88 /dev/null 2024-09-16T16:34:32.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:32.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj89 /dev/null 2024-09-16T16:34:32.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:32.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj90 /dev/null 2024-09-16T16:34:32.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:32.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj91 /dev/null 2024-09-16T16:34:32.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:32.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj92 /dev/null 2024-09-16T16:34:33.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:33.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj93 /dev/null 2024-09-16T16:34:33.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:33.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj94 /dev/null 2024-09-16T16:34:33.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:33.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj95 /dev/null 2024-09-16T16:34:33.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:33.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj96 /dev/null 2024-09-16T16:34:33.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:33.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj97 /dev/null 2024-09-16T16:34:33.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:33.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj98 /dev/null 2024-09-16T16:34:34.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:34.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj99 /dev/null 2024-09-16T16:34:34.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:34.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj100 /dev/null 2024-09-16T16:34:34.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:34.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj101 /dev/null 2024-09-16T16:34:34.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:34.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj102 /dev/null 2024-09-16T16:34:34.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:34.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj103 /dev/null 2024-09-16T16:34:35.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:35.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj104 /dev/null 2024-09-16T16:34:35.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:35.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj105 /dev/null 2024-09-16T16:34:35.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:35.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj106 /dev/null 2024-09-16T16:34:35.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:35.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj107 /dev/null 2024-09-16T16:34:35.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:35.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj108 /dev/null 2024-09-16T16:34:36.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:36.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj109 /dev/null 2024-09-16T16:34:36.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:36.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj110 /dev/null 2024-09-16T16:34:36.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:36.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj111 /dev/null 2024-09-16T16:34:36.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:36.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj112 /dev/null 2024-09-16T16:34:36.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:36.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj113 /dev/null 2024-09-16T16:34:37.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:37.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj114 /dev/null 2024-09-16T16:34:37.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:37.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj115 /dev/null 2024-09-16T16:34:37.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:37.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj116 /dev/null 2024-09-16T16:34:37.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:37.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj117 /dev/null 2024-09-16T16:34:37.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:37.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj118 /dev/null 2024-09-16T16:34:38.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:38.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj119 /dev/null 2024-09-16T16:34:38.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:38.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj120 /dev/null 2024-09-16T16:34:38.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:38.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj121 /dev/null 2024-09-16T16:34:38.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:38.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj122 /dev/null 2024-09-16T16:34:38.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:38.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj123 /dev/null 2024-09-16T16:34:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj124 /dev/null 2024-09-16T16:34:39.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:39.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj125 /dev/null 2024-09-16T16:34:39.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:39.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj126 /dev/null 2024-09-16T16:34:39.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:39.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj127 /dev/null 2024-09-16T16:34:39.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:39.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj128 /dev/null 2024-09-16T16:34:39.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:39.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj129 /dev/null 2024-09-16T16:34:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj130 /dev/null 2024-09-16T16:34:40.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:40.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj131 /dev/null 2024-09-16T16:34:40.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:40.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj132 /dev/null 2024-09-16T16:34:40.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:40.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj133 /dev/null 2024-09-16T16:34:40.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:40.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj134 /dev/null 2024-09-16T16:34:41.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:41.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj135 /dev/null 2024-09-16T16:34:41.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:41.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj136 /dev/null 2024-09-16T16:34:41.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:41.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj137 /dev/null 2024-09-16T16:34:41.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:41.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj138 /dev/null 2024-09-16T16:34:42.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:42.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj139 /dev/null 2024-09-16T16:34:42.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:42.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj140 /dev/null 2024-09-16T16:34:42.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:42.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj141 /dev/null 2024-09-16T16:34:42.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:42.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj142 /dev/null 2024-09-16T16:34:42.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:42.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj143 /dev/null 2024-09-16T16:34:42.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:42.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj144 /dev/null 2024-09-16T16:34:43.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:43.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj145 /dev/null 2024-09-16T16:34:43.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:43.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj146 /dev/null 2024-09-16T16:34:43.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:43.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj147 /dev/null 2024-09-16T16:34:43.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:43.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj148 /dev/null 2024-09-16T16:34:43.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:43.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj149 /dev/null 2024-09-16T16:34:44.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:44.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj150 /dev/null 2024-09-16T16:34:44.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:44.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj151 /dev/null 2024-09-16T16:34:44.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:44.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj152 /dev/null 2024-09-16T16:34:44.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:44.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj153 /dev/null 2024-09-16T16:34:44.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:44.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj154 /dev/null 2024-09-16T16:34:44.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:44.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj155 /dev/null 2024-09-16T16:34:45.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:45.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj156 /dev/null 2024-09-16T16:34:45.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:45.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj157 /dev/null 2024-09-16T16:34:45.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:45.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj158 /dev/null 2024-09-16T16:34:45.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:45.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj159 /dev/null 2024-09-16T16:34:46.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:46.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj160 /dev/null 2024-09-16T16:34:46.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:46.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj161 /dev/null 2024-09-16T16:34:46.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:46.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj162 /dev/null 2024-09-16T16:34:46.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:46.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj163 /dev/null 2024-09-16T16:34:46.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:46.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj164 /dev/null 2024-09-16T16:34:46.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:46.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj165 /dev/null 2024-09-16T16:34:47.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:47.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj166 /dev/null 2024-09-16T16:34:47.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:47.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj167 /dev/null 2024-09-16T16:34:47.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:47.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj168 /dev/null 2024-09-16T16:34:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:47.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj169 /dev/null 2024-09-16T16:34:47.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:47.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj170 /dev/null 2024-09-16T16:34:48.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:48.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj171 /dev/null 2024-09-16T16:34:48.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:48.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj172 /dev/null 2024-09-16T16:34:48.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:48.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj173 /dev/null 2024-09-16T16:34:48.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:48.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj174 /dev/null 2024-09-16T16:34:48.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:48.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj175 /dev/null 2024-09-16T16:34:49.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:49.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj176 /dev/null 2024-09-16T16:34:49.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:49.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj177 /dev/null 2024-09-16T16:34:49.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:49.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj178 /dev/null 2024-09-16T16:34:49.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:49.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj179 /dev/null 2024-09-16T16:34:49.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:49.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj180 /dev/null 2024-09-16T16:34:50.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:50.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj181 /dev/null 2024-09-16T16:34:50.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:50.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj182 /dev/null 2024-09-16T16:34:50.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:50.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj183 /dev/null 2024-09-16T16:34:50.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:50.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj184 /dev/null 2024-09-16T16:34:50.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:50.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj185 /dev/null 2024-09-16T16:34:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:51.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj186 /dev/null 2024-09-16T16:34:51.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:51.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj187 /dev/null 2024-09-16T16:34:51.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:51.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj188 /dev/null 2024-09-16T16:34:51.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:51.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj189 /dev/null 2024-09-16T16:34:52.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:52.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj190 /dev/null 2024-09-16T16:34:52.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:52.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj191 /dev/null 2024-09-16T16:34:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:52.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj192 /dev/null 2024-09-16T16:34:52.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:52.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj193 /dev/null 2024-09-16T16:34:52.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:52.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj194 /dev/null 2024-09-16T16:34:53.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:53.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj195 /dev/null 2024-09-16T16:34:53.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:53.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj196 /dev/null 2024-09-16T16:34:53.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:53.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj197 /dev/null 2024-09-16T16:34:53.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:53.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj198 /dev/null 2024-09-16T16:34:53.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:53.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj199 /dev/null 2024-09-16T16:34:54.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-16T16:34:54.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj200 /dev/null 2024-09-16T16:34:54.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: get_primary test obj1 2024-09-16T16:34:54.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:34:54.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:34:54.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:34:54.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:34:54.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: local primary=1 2024-09-16T16:34:54.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: get_pg test obj1 2024-09-16T16:34:54.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-16T16:34:54.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-16T16:34:54.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-16T16:34:54.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T16:34:55.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: local PG=1.0 2024-09-16T16:34:55.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: get_not_primary test obj1 2024-09-16T16:34:55.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-16T16:34:55.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-16T16:34:55.188 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-16T16:34:55.188 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:34:55.188 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:34:55.188 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:34:55.189 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:34:55.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-16T16:34:55.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-16T16:34:55.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-16T16:34:56.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: local otherosd=0 2024-09-16T16:34:56.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:408: TEST_recovery_erasure_remapped: ceph osd set norecover 2024-09-16T16:34:57.067 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is set 2024-09-16T16:34:57.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: cat td/osd-recovery-stats/osd.0.pid 2024-09-16T16:34:57.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: kill 292387 2024-09-16T16:34:57.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:410: TEST_recovery_erasure_remapped: ceph osd down osd.0 2024-09-16T16:34:57.881 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T16:34:57.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:411: TEST_recovery_erasure_remapped: ceph osd out osd.0 2024-09-16T16:34:58.635 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T16:34:58.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-09-16T16:34:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 0 = 1 ']' 2024-09-16T16:34:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 0 = 0 ']' 2024-09-16T16:34:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:422: TEST_recovery_erasure_remapped: continue 2024-09-16T16:34:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-09-16T16:34:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 1 = 1 ']' 2024-09-16T16:34:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:418: TEST_recovery_erasure_remapped: continue 2024-09-16T16:34:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-09-16T16:34:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 2 = 1 ']' 2024-09-16T16:34:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 2 = 0 ']' 2024-09-16T16:34:58.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:424: TEST_recovery_erasure_remapped: ceph osd out osd.2 2024-09-16T16:34:59.309 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already out. 2024-09-16T16:34:59.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:425: TEST_recovery_erasure_remapped: break 2024-09-16T16:34:59.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:427: TEST_recovery_erasure_remapped: ceph osd unset norecover 2024-09-16T16:35:00.132 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is unset 2024-09-16T16:35:00.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: get_primary test obj1 2024-09-16T16:35:00.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:35:00.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:35:00.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:35:00.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:35:00.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-16T16:35:00.838 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T16:35:00.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:429: TEST_recovery_erasure_remapped: sleep 2 2024-09-16T16:35:02.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:431: TEST_recovery_erasure_remapped: wait_for_clean 2024-09-16T16:35:02.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:35:02.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:35:02.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:35:02.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:35:02.855 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:35:02.856 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:35:02.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:35:02.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:35:02.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:35:02.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:35:02.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:35:02.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:35:02.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:35:02.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:35:02.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:35:03.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:35:03.423 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:35:03.424 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:35:03.424 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T16:35:03.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:35:03.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:35:03.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:35:03.534 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-16T16:35:03.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T16:35:03.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T16:35:03.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T16:35:03.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:35:03.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:35:03.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672994 2024-09-16T16:35:03.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672994 2024-09-16T16:35:03.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949672994' 2024-09-16T16:35:03.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:35:03.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:35:03.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509470 2024-09-16T16:35:03.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509470 2024-09-16T16:35:03.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949672994 2-64424509470' 2024-09-16T16:35:03.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:35:03.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:35:04.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345939 2024-09-16T16:35:04.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345939 2024-09-16T16:35:04.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949672994 2-64424509470 3-85899345939' 2024-09-16T16:35:04.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:35:04.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:35:04.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672994 2024-09-16T16:35:04.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:35:04.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672994 2024-09-16T16:35:04.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:35:04.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672994 2024-09-16T16:35:04.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-09-16T16:35:04.003 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672994 2024-09-16T16:35:04.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:35:04.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672992 -lt 42949672994 2024-09-16T16:35:04.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:35:05.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:35:05.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:35:05.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672994 -lt 42949672994 2024-09-16T16:35:05.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:35:05.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509470 2024-09-16T16:35:05.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:35:05.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:35:05.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509470 2024-09-16T16:35:05.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:35:05.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509470 2024-09-16T16:35:05.972 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509470 2024-09-16T16:35:05.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-09-16T16:35:05.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:35:06.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509471 -lt 64424509470 2024-09-16T16:35:06.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:35:06.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345939 2024-09-16T16:35:06.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:35:06.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:35:06.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345939 2024-09-16T16:35:06.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:35:06.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345939 2024-09-16T16:35:06.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2024-09-16T16:35:06.486 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345939 2024-09-16T16:35:06.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:35:07.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345940 -lt 85899345939 2024-09-16T16:35:07.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:35:07.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:35:07.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:35:07.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:35:07.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:35:07.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:35:07.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:35:07.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:35:07.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:35:07.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:35:07.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:35:08.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:35:08.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:35:08.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:35:08.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:35:08.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:35:08.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T16:35:08.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T16:35:08.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T16:35:08.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:35:08.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:35:09.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:35:09.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:35:09.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:35:09.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:35:09.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:35:09.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:35:09.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:35:09.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:35:09.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:35:09.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:35:09.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:35:09.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:35:10.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:35:10.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:35:10.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:35:10.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:35:10.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:35:10.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:35:10.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:35:10.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:35:10.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:35:10.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-09-16T16:35:10.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-09-16T16:35:10.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:35:10.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:35:10.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:35:10.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:35:10.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:35:10.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:35:10.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:35:10.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:35:10.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:35:10.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:35:10.916 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:35:11.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:35:11.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:35:11.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:35:11.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:35:12.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:35:12.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:35:12.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:35:12.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:35:12.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:35:12.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:35:12.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:35:12.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:35:12.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:35:12.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14 2024-09-16T16:35:12.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14 '!=' null 2024-09-16T16:35:12.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:35:12.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:35:12.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:35:12.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:35:12.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:35:12.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:35:12.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:35:12.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:35:12.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:35:12.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:35:12.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:35:13.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:35:13.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:35:13.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:35:13.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:35:13.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:35:13.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:35:13.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:35:13.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:35:13.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:35:13.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:35:13.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:35:13.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:35:13.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:35:14.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12 2024-09-16T16:35:14.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12 '!=' null 2024-09-16T16:35:14.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:35:14.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:35:14.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:35:14.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:35:14.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:35:14.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:35:14.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:35:14.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:35:14.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:35:14.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:35:14.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:35:15.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:35:15.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:35:15.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:35:15.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:35:15.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:35:15.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:35:15.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:35:15.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:35:15.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:35:15.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:35:15.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:35:15.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:35:15.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:35:16.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-16T16:35:16.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-16T16:35:16.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:35:16.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:35:16.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:35:16.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:35:16.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:35:16.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:35:16.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:35:16.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:35:16.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:35:16.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:35:16.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:35:16.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:35:16.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:35:16.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:35:16.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:35:17.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:35:17.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:35:17.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:35:17.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:35:17.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:35:17.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:35:17.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:35:17.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:35:17.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:35:17.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16 2024-09-16T16:35:17.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16 '!=' null 2024-09-16T16:35:17.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:35:17.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:35:17.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:35:17.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:35:17.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:35:17.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:35:17.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:35:17.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:35:17.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:35:17.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:35:17.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:35:18.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:35:18.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:35:18.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:35:18.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:35:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:35:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:35:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:35:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:433: TEST_recovery_erasure_remapped: local log=td/osd-recovery-stats/osd.1.log 2024-09-16T16:35:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:434: TEST_recovery_erasure_remapped: check td/osd-recovery-stats 1.0 1 erasure 200 0 200 200 2024-09-16T16:35:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-16T16:35:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-16T16:35:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-16T16:35:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-09-16T16:35:18.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=200 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=200 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-16T16:35:18.928 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-16T16:35:18.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,NONE]/[1,NONE,2]' 2024-09-16T16:35:18.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-16T16:35:18.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:35:18.945 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:35:18.945 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-09-16T16:35:18.945 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-16T16:35:18.945 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-16T16:35:18.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:35:18.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-09-16T16:35:18.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-09-16T16:35:18.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-16T16:35:18.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:35:18.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-16T16:35:18.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:35:18.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:35:18.956 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:35:18.956 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-16T16:35:18.956 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:35:18.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-16T16:35:18.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-16T16:35:18.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:35:18.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:35:18.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:35:18.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:35:18.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-16T16:35:18.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:35:18.969 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:35:18.969 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-09-16T16:35:18.969 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-16T16:35:18.969 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-16T16:35:18.969 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:35:18.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=200 2024-09-16T16:35:18.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 200 200 2024-09-16T16:35:18.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-16T16:35:18.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:35:18.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-16T16:35:18.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:35:18.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:35:18.979 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:35:18.979 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-16T16:35:18.979 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:35:18.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=200 2024-09-16T16:35:18.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 200 200 2024-09-16T16:35:18.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=200 2024-09-16T16:35:18.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:35:18.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=200 2024-09-16T16:35:18.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:35:18.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-16T16:35:18.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:436: TEST_recovery_erasure_remapped: delete_pool test 2024-09-16T16:35:18.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T16:35:18.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T16:35:19.612 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T16:35:19.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:437: TEST_recovery_erasure_remapped: kill_daemons td/osd-recovery-stats 2024-09-16T16:35:19.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:35:19.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:35:19.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:35:19.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:35:19.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:35:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:35:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-16T16:35:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:35:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:35:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:35:24.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:35:24.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:35:24.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:35:24.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:35:24.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:35:24.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:35:24.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:35:24.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:35:24.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:35:24.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:35:24.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:35:24.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:35:24.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:35:24.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:35:24.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:35:24.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:35:24.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:35:24.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:35:24.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:35:24.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:35:24.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:35:24.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:35:24.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:35:24.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:35:24.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:35:25.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:35:25.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:35:25.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:35:25.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:35:25.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:35:25.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:35:25.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:35:25.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:35:25.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:35:25.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:35:25.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:35:25.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:35:25.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:35:25.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:35:25.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:35:25.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:35:25.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:35:25.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:35:25.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:35:25.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:35:25.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-16T16:35:25.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:35:25.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:35:25.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:35:25.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274941 2024-09-16T16:35:25.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:35:25.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_multi td/osd-recovery-stats 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:441: TEST_recovery_multi: local dir=td/osd-recovery-stats 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:443: TEST_recovery_multi: local osds=6 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:444: TEST_recovery_multi: run_mon td/osd-recovery-stats a 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-16T16:35:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-16T16:35:25.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:35:25.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:35:25.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:35:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:35:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:35:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:35:25.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:35:25.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:35:25.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:35:25.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:35:25.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:35:25.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:35:25.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:35:25.633 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:35:25.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:35:25.633 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:35:25.633 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:35:25.634 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:35:25.634 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:35:25.634 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:35:25.634 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:35:25.634 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:35:25.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get fsid 2024-09-16T16:35:25.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:35:25.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:35:25.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:35:25.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:35:25.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:35:25.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:35:25.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:35:25.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:35:25.756 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:35:25.757 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:35:25.757 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:35:25.757 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:35:25.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:35:25.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get mon_host 2024-09-16T16:35:25.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:445: TEST_recovery_multi: run_mgr td/osd-recovery-stats x 2024-09-16T16:35:25.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-16T16:35:25.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:35:25.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:35:25.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:35:25.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-16T16:35:25.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:35:26.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:35:26.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:35:26.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:35:26.790 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:35:26.790 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:35:26.790 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:35:26.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:35:26.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:35:26.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:35:26.860 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: expr 6 - 1 2024-09-16T16:35:26.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: seq 0 5 2024-09-16T16:35:26.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-16T16:35:26.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 0 2024-09-16T16:35:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:35:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:35:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:35:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:35:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-16T16:35:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:35:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:35:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:35:26.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:35:26.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-16T16:35:26.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-16T16:35:26.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:35:26.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:35:26.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:35:26.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:35:26.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:35:26.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:35:26.868 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:35:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:35:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:35:26.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:35:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:35:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:35:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:35:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:35:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:35:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:35:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:35:26.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:35:26.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:35:26.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:35:26.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:35:26.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:35:26.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-16T16:35:26.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:35:26.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8a474966-f593-4873-9315-ee6384b1da9e 2024-09-16T16:35:26.871 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 8a474966-f593-4873-9315-ee6384b1da9e 2024-09-16T16:35:26.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8a474966-f593-4873-9315-ee6384b1da9e' 2024-09-16T16:35:26.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:35:26.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBOXuhmxxi3NBAA/tmoiZfZrSroZYB5GdDBmg== 2024-09-16T16:35:26.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBOXuhmxxi3NBAA/tmoiZfZrSroZYB5GdDBmg=="}' 2024-09-16T16:35:26.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8a474966-f593-4873-9315-ee6384b1da9e -i td/osd-recovery-stats/0/new.json 2024-09-16T16:35:27.225 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:35:27.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-16T16:35:27.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOXuhmxxi3NBAA/tmoiZfZrSroZYB5GdDBmg== --osd-uuid 8a474966-f593-4873-9315-ee6384b1da9e 2024-09-16T16:35:27.271 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:27.269+0000 7fa0582c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:35:27.284 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:27.281+0000 7fa0582c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:35:27.286 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:27.285+0000 7fa0582c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:35:27.286 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:27.285+0000 7fa0582c57c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:35:27.287 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:27.285+0000 7fa0582c57c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:35:27.287 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:27.285+0000 7fa0582c57c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:35:27.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:27.313+0000 7fa0582c57c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-16T16:35:33.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-16T16:35:33.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:35:33.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:35:33.441 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:35:33.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:35:34.020 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:35:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:35:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:35:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:35:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:35:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:35:34.053 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:34.049+0000 7f5fc520f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:35:34.082 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:34.081+0000 7f5fc520f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:35:34.083 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:34.081+0000 7f5fc520f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:35:34.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:35:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:35:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:35:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:35:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:35:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:35:34.472 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:35:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:34.643 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:34.641+0000 7f5fc520f7c0 -1 Falling back to public interface 2024-09-16T16:35:34.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:35.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:35.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:35.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:35:35.899 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:35:35.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:35.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:36.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:37.231 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:35:37.229+0000 7f5fc520f7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:35:37.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:37.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:37.318 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:35:37.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:35:37.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:37.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:37.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:38.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:38.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:38.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:35:38.898 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:35:38.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:38.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:39.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:40.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:40.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:40.368 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:35:40.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:35:40.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:40.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:40.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:41.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:41.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:41.983 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:35:41.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:35:41.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:41.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:42.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:43.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:43.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:43.575 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:35:43.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:35:43.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:43.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:44.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:45.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:45.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:45.068 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:35:45.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:35:45.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:45.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:45.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:46.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:46.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:46.549 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:35:46.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:35:46.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:46.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:47.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:48.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:48.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:48.121 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:35:48.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:35:48.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:48.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:48.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:49.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:49.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:49.596 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:35:49.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:35:49.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:49.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:50.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:51.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:51.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:51.478 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:35:51.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:35:51.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:51.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:52.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:53.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:53.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:53.407 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:35:53.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:35:53.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:53.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:53.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:54.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:54.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:54.995 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:35:54.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:35:54.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:54.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:55.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:56.500 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:35:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:35:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:56.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:57.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:35:58.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:35:58.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:35:58.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:35:58.289 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:35:58.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:35:58.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:35:59.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:00.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:00.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:00.387 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:36:00.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:36:00.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:00.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:36:00.821 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3242837029,v1:127.0.0.1:6803/3242837029] [v2:127.0.0.1:6804/3242837029,v1:127.0.0.1:6805/3242837029] exists,up 8a474966-f593-4873-9315-ee6384b1da9e 2024-09-16T16:36:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:36:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:36:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:36:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-16T16:36:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 1 2024-09-16T16:36:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:36:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:36:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:36:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:36:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-16T16:36:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:36:00.825 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:36:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-16T16:36:00.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:36:00.827 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 e8b8a2b4-8e30-4c8c-8e29-496baf8755bf 2024-09-16T16:36:00.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e8b8a2b4-8e30-4c8c-8e29-496baf8755bf 2024-09-16T16:36:00.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e8b8a2b4-8e30-4c8c-8e29-496baf8755bf' 2024-09-16T16:36:00.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:36:00.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwXuhm4qSEMhAAys353tuiDb3nKcIUCZFDgQ== 2024-09-16T16:36:00.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwXuhm4qSEMhAAys353tuiDb3nKcIUCZFDgQ=="}' 2024-09-16T16:36:00.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e8b8a2b4-8e30-4c8c-8e29-496baf8755bf -i td/osd-recovery-stats/1/new.json 2024-09-16T16:36:01.384 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:36:01.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-16T16:36:01.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwXuhm4qSEMhAAys353tuiDb3nKcIUCZFDgQ== --osd-uuid e8b8a2b4-8e30-4c8c-8e29-496baf8755bf 2024-09-16T16:36:01.430 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:01.424+0000 7ff58df617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:01.432 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:01.428+0000 7ff58df617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:01.433 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:01.428+0000 7ff58df617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:01.433 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:01.428+0000 7ff58df617c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:36:01.433 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:01.428+0000 7ff58df617c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:36:01.433 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:01.428+0000 7ff58df617c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:36:01.515 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:01.512+0000 7ff58df617c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-16T16:36:07.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-16T16:36:07.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:36:07.660 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:36:07.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:36:07.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:36:08.232 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:36:08.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:36:08.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:36:08.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:36:08.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:36:08.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:36:08.266 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:08.264+0000 7f913ba467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:08.268 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:08.264+0000 7f913ba467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:08.270 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:08.268+0000 7f913ba467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:08.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:36:08.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:36:08.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:36:08.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:36:08.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:36:08.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:08.643 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:36:08.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:36:08.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:08.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:08.819 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:08.816+0000 7f913ba467c0 -1 Falling back to public interface 2024-09-16T16:36:09.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:10.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:10.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:10.071 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:36:10.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:36:10.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:10.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:11.422 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:11.420+0000 7f913ba467c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:36:11.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:11.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:11.548 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:36:11.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:36:11.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:11.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:12.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:13.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:13.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:13.179 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:36:13.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:36:13.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:13.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:13.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:14.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:14.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:14.633 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:36:14.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:36:14.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:14.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:15.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:16.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:16.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:16.075 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:36:16.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:36:16.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:16.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:16.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:17.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:17.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:17.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:36:17.655 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:36:17.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:17.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:18.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:19.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:19.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:19.150 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:36:19.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:36:19.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:19.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:19.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:20.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:20.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:20.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:36:20.590 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:36:20.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:20.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:21.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:22.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:22.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:22.027 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:36:22.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:36:22.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:22.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:22.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:23.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:23.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:23.569 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:36:23.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:36:23.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:23.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:24.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:25.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:25.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:25.165 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:36:25.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:36:25.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:25.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:25.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:26.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:26.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:26.701 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:36:26.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:36:26.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:26.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:27.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:28.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:28.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:28.413 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:36:28.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:36:28.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:28.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:29.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:30.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:30.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:30.154 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:36:30.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:36:30.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:30.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:31.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:32.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:32.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:32.032 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:36:32.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:36:32.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:32.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:32.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:33.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:33.836 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:36:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:36:33.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:33.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:34.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:35.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:35.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:35.554 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:36:35.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:36:35.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:35.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:36:36.078 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/642590606,v1:127.0.0.1:6811/642590606] [v2:127.0.0.1:6812/642590606,v1:127.0.0.1:6813/642590606] exists,up e8b8a2b4-8e30-4c8c-8e29-496baf8755bf 2024-09-16T16:36:36.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 2 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:36:36.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:36:36.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:36:36.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-16T16:36:36.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-16T16:36:36.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:36:36.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:36:36.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:36:36.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:36:36.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:36:36.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:36:36.084 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:36:36.084 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:36:36.084 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:36:36.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:36:36.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-16T16:36:36.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:36:36.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d6859e43-7f64-4d27-bef9-7220930e6598 2024-09-16T16:36:36.086 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 d6859e43-7f64-4d27-bef9-7220930e6598 2024-09-16T16:36:36.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d6859e43-7f64-4d27-bef9-7220930e6598' 2024-09-16T16:36:36.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:36:36.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUXuhm8CDfBRAAIaBBbpal/J4rLYo8oF7LDQ== 2024-09-16T16:36:36.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUXuhm8CDfBRAAIaBBbpal/J4rLYo8oF7LDQ=="}' 2024-09-16T16:36:36.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d6859e43-7f64-4d27-bef9-7220930e6598 -i td/osd-recovery-stats/2/new.json 2024-09-16T16:36:36.650 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:36:36.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-16T16:36:36.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUXuhm8CDfBRAAIaBBbpal/J4rLYo8oF7LDQ== --osd-uuid d6859e43-7f64-4d27-bef9-7220930e6598 2024-09-16T16:36:36.704 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:36.699+0000 7fddf828a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:36.706 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:36.703+0000 7fddf828a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:36.707 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:36.703+0000 7fddf828a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:36.708 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:36.703+0000 7fddf828a7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:36:36.708 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:36.703+0000 7fddf828a7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:36:36.708 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:36.703+0000 7fddf828a7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:36:36.773 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:36.767+0000 7fddf828a7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-16T16:36:43.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-16T16:36:43.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:36:43.236 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T16:36:43.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T16:36:43.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:36:43.825 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T16:36:43.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T16:36:43.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:36:43.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:36:43.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:36:43.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:36:43.853 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:43.847+0000 7f34670ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:43.874 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:43.871+0000 7f34670ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:43.875 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:43.871+0000 7f34670ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:36:44.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T16:36:44.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:36:44.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T16:36:44.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:36:44.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:36:44.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:44.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:36:44.262 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:36:44.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:44.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:36:44.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:44.455+0000 7f34670ee7c0 -1 Falling back to public interface 2024-09-16T16:36:44.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:45.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:45.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:45.727 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:36:45.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:36:45.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:45.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:36:46.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:47.002 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:36:46.999+0000 7f34670ee7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T16:36:47.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:47.152 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:36:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:36:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:47.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:36:47.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:36:48.770 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:36:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:48.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:36:49.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:50.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:50.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:50.204 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:36:50.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:36:50.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:50.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:36:50.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:51.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:51.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:51.679 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:36:51.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:36:51.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:51.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:36:52.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:53.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:53.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:53.352 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:36:53.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:36:53.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:53.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:36:54.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:55.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:55.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:55.064 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:36:55.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:36:55.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:55.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:36:55.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:56.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:56.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:56.558 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:36:56.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:36:56.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:56.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:36:57.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:58.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:58.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:58.194 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:36:58.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:36:58.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:58.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:36:58.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:36:59.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:36:59.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:36:59.827 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:36:59.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:36:59.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:36:59.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:37:00.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:01.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:01.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:01.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:37:01.353 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:37:01.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:01.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:37:01.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:02.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:02.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:02.873 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:37:02.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:37:02.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:02.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:37:03.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:04.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:04.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:04.554 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:37:04.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:37:04.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:04.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:37:05.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:06.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:06.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:06.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:37:06.596 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:37:06.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:06.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:37:07.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:08.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:08.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:08.531 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:37:08.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:37:08.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:08.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:37:09.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:10.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:10.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:10.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:37:10.243 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:37:10.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:10.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:37:11.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:12.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:12.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:12.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:37:12.011 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:37:12.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:12.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:37:12.442 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2566459792,v1:127.0.0.1:6819/2566459792] [v2:127.0.0.1:6820/2566459792,v1:127.0.0.1:6821/2566459792] exists,up d6859e43-7f64-4d27-bef9-7220930e6598 2024-09-16T16:37:12.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:37:12.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:37:12.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:37:12.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-16T16:37:12.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 3 2024-09-16T16:37:12.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:37:12.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:37:12.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:37:12.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:37:12.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:37:12.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:37:12.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:37:12.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-16T16:37:12.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:37:12.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=16fc655d-e7a7-4f3d-9fb3-30d0e6edc163 2024-09-16T16:37:12.450 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 16fc655d-e7a7-4f3d-9fb3-30d0e6edc163 2024-09-16T16:37:12.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 16fc655d-e7a7-4f3d-9fb3-30d0e6edc163' 2024-09-16T16:37:12.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:37:12.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4XuhmMJEzHBAANWCZmFg+/9iL8E7yjZ+SUA== 2024-09-16T16:37:12.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4XuhmMJEzHBAANWCZmFg+/9iL8E7yjZ+SUA=="}' 2024-09-16T16:37:12.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 16fc655d-e7a7-4f3d-9fb3-30d0e6edc163 -i td/osd-recovery-stats/3/new.json 2024-09-16T16:37:13.036 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:37:13.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-16T16:37:13.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4XuhmMJEzHBAANWCZmFg+/9iL8E7yjZ+SUA== --osd-uuid 16fc655d-e7a7-4f3d-9fb3-30d0e6edc163 2024-09-16T16:37:13.085 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:13.083+0000 7f44dae807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:13.085 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:13.083+0000 7f44dae807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:13.087 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:13.087+0000 7f44dae807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:13.087 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:13.087+0000 7f44dae807c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:37:13.087 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:13.087+0000 7f44dae807c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:37:13.087 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:13.087+0000 7f44dae807c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:37:13.174 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:13.175+0000 7f44dae807c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-16T16:37:19.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-16T16:37:19.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:37:19.335 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T16:37:19.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T16:37:19.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:37:19.922 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T16:37:19.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T16:37:19.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:37:19.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:37:19.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:37:19.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:37:19.957 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:19.955+0000 7f8cc6f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:19.959 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:19.959+0000 7f8cc6f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:19.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:19.959+0000 7f8cc6f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:20.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T16:37:20.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:37:20.388 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:37:20.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T16:37:20.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:37:20.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:37:20.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:20.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:37:20.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:20.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:20.519 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:20.519+0000 7f8cc6f0d7c0 -1 Falling back to public interface 2024-09-16T16:37:20.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:21.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:21.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:21.873 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:37:21.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:37:21.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:21.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:22.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:23.101 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:23.098+0000 7f8cc6f0d7c0 -1 osd.3 0 log_to_monitors true 2024-09-16T16:37:23.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:23.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:23.375 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:37:23.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:37:23.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:23.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:23.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:24.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:24.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:24.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:37:24.970 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:37:24.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:24.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:25.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:26.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:26.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:26.402 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:37:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:37:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:26.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:27.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:27.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:27.872 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:37:27.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:37:27.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:27.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:28.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:29.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:29.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:29.333 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:37:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:37:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:29.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:29.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:30.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:30.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:30.948 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:37:30.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:37:30.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:30.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:31.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:32.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:32.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:32.379 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:37:32.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:37:32.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:32.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:33.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:33.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:33.889 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:37:33.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:37:33.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:33.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:34.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:35.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:35.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:35.420 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:37:35.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:37:35.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:35.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:36.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:37.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:37.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:37.006 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:37:37.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:37:37.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:37.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:37.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:38.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:38.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:38.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:37:38.664 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:37:38.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:38.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:39.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:40.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:40.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:40.385 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:37:40.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:37:40.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:40.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:41.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:42.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:42.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:42.411 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:37:42.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:37:42.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:42.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:43.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:44.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:44.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:44.246 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:37:44.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:37:44.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:44.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:45.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:46.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:46.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:46.350 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:37:46.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:37:46.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:46.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:37:46.773 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2755797696,v1:127.0.0.1:6827/2755797696] [v2:127.0.0.1:6828/2755797696,v1:127.0.0.1:6829/2755797696] exists,up 16fc655d-e7a7-4f3d-9fb3-30d0e6edc163 2024-09-16T16:37:46.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:37:46.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:37:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:37:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-16T16:37:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 4 2024-09-16T16:37:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:37:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:37:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T16:37:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:37:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-16T16:37:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:37:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:37:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:37:46.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:37:46.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-16T16:37:46.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-16T16:37:46.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:37:46.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:37:46.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:37:46.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:37:46.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:37:46.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:37:46.780 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:37:46.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:37:46.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:37:46.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:37:46.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-16T16:37:46.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:37:46.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50383f6b-7d2e-442e-a774-5eb477a60296 2024-09-16T16:37:46.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 50383f6b-7d2e-442e-a774-5eb477a60296' 2024-09-16T16:37:46.784 INFO:tasks.workunit.client.0.smithi039.stdout:add osd4 50383f6b-7d2e-442e-a774-5eb477a60296 2024-09-16T16:37:46.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:37:46.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDaXuhm8qpkLxAAt1sahtO1dorg8ZUoZmylXg== 2024-09-16T16:37:46.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDaXuhm8qpkLxAAt1sahtO1dorg8ZUoZmylXg=="}' 2024-09-16T16:37:46.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50383f6b-7d2e-442e-a774-5eb477a60296 -i td/osd-recovery-stats/4/new.json 2024-09-16T16:37:47.354 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:37:47.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-16T16:37:47.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaXuhm8qpkLxAAt1sahtO1dorg8ZUoZmylXg== --osd-uuid 50383f6b-7d2e-442e-a774-5eb477a60296 2024-09-16T16:37:47.407 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:47.406+0000 7f6d96e207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:47.433 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:47.430+0000 7f6d96e207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:47.434 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:47.434+0000 7f6d96e207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:47.434 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:47.434+0000 7f6d96e207c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:37:47.435 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:47.434+0000 7f6d96e207c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:37:47.435 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:47.434+0000 7f6d96e207c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:37:47.550 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:47.550+0000 7f6d96e207c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-16T16:37:53.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-16T16:37:53.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:37:53.748 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd4 key to auth repository 2024-09-16T16:37:53.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T16:37:53.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:37:54.313 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.4 2024-09-16T16:37:54.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T16:37:54.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:37:54.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:37:54.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:37:54.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:37:54.339 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:54.338+0000 7f40557517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:54.341 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:54.338+0000 7f40557517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:54.342 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:54.342+0000 7f40557517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:37:54.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T16:37:54.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:37:54.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T16:37:54.763 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:37:54.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:37:54.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:37:54.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:54.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:37:54.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:54.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:37:54.891 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:54.890+0000 7f40557517c0 -1 Falling back to public interface 2024-09-16T16:37:55.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:56.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:56.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:56.218 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:37:56.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:37:56.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:56.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:37:56.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:57.516 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:37:57.514+0000 7f40557517c0 -1 osd.4 0 log_to_monitors true 2024-09-16T16:37:57.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:57.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:57.637 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:37:57.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:37:57.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:57.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:37:58.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:37:59.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:37:59.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:37:59.281 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:37:59.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:37:59.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:37:59.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:37:59.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:00.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:00.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:00.713 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:38:00.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:38:00.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:00.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:01.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:02.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:02.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:02.160 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:38:02.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:38:02.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:02.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:02.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:03.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:03.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:03.636 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:38:03.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:38:03.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:03.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:04.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:05.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:05.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:05.241 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:38:05.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:38:05.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:05.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:05.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:06.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:06.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:06.703 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:38:06.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:38:06.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:06.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:07.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:08.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:08.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:08.141 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:38:08.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:38:08.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:08.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:08.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:09.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:09.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:09.664 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:38:09.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:38:09.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:09.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:10.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:11.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:11.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:11.249 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:38:11.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:38:11.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:11.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:12.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:13.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:13.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:13.027 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:38:13.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:38:13.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:13.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:13.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:14.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:14.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:14.729 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:38:14.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:38:14.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:14.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:15.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:16.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:16.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:16.485 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:38:16.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:38:16.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:16.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:17.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:18.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:18.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:18.544 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:38:18.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:38:18.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:18.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:19.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:20.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:20.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:20.484 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:38:20.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:38:20.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:20.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:21.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:22.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:22.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:22.434 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:38:22.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:38:22.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:22.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:38:22.969 INFO:tasks.workunit.client.0.smithi039.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3046049527,v1:127.0.0.1:6835/3046049527] [v2:127.0.0.1:6836/3046049527,v1:127.0.0.1:6837/3046049527] exists,up 50383f6b-7d2e-442e-a774-5eb477a60296 2024-09-16T16:38:22.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:38:22.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:38:22.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:38:22.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 5 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-16T16:38:22.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:38:22.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:38:22.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:38:22.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:38:22.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:38:22.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:38:22.974 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:38:22.974 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:38:22.974 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:38:22.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:38:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:38:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:38:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:38:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:38:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:38:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:38:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:38:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:38:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:38:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:38:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:38:22.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:38:22.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-16T16:38:22.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:38:22.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=870d170c-df6e-4670-bb43-1861ba08b2f4 2024-09-16T16:38:22.977 INFO:tasks.workunit.client.0.smithi039.stdout:add osd5 870d170c-df6e-4670-bb43-1861ba08b2f4 2024-09-16T16:38:22.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 870d170c-df6e-4670-bb43-1861ba08b2f4' 2024-09-16T16:38:22.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:38:22.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+Xuhmx5yBOxAANUd7kxE04Qugtui9+OkQ+Q== 2024-09-16T16:38:22.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+Xuhmx5yBOxAANUd7kxE04Qugtui9+OkQ+Q=="}' 2024-09-16T16:38:22.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 870d170c-df6e-4670-bb43-1861ba08b2f4 -i td/osd-recovery-stats/5/new.json 2024-09-16T16:38:23.617 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:38:23.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-16T16:38:23.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+Xuhmx5yBOxAANUd7kxE04Qugtui9+OkQ+Q== --osd-uuid 870d170c-df6e-4670-bb43-1861ba08b2f4 2024-09-16T16:38:23.668 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:23.665+0000 7f7ae06037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:38:23.694 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:23.693+0000 7f7ae06037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:38:23.695 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:23.693+0000 7f7ae06037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:38:23.696 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:23.693+0000 7f7ae06037c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:38:23.696 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:23.693+0000 7f7ae06037c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:38:23.696 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:23.693+0000 7f7ae06037c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:38:23.785 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:23.785+0000 7f7ae06037c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-16T16:38:29.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-16T16:38:29.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:38:29.828 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd5 key to auth repository 2024-09-16T16:38:29.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T16:38:29.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:38:30.407 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.5 2024-09-16T16:38:30.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T16:38:30.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:38:30.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:38:30.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:38:30.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:38:30.441 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:30.437+0000 7fa033e1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:38:30.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:30.457+0000 7fa033e1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:38:30.462 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:30.461+0000 7fa033e1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:38:30.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T16:38:30.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:38:30.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T16:38:30.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:38:30.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:38:30.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:30.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:38:30.839 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:38:30.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:30.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:31.031 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:31.029+0000 7fa033e1f7c0 -1 Falling back to public interface 2024-09-16T16:38:31.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:32.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:32.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:32.293 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:38:32.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:38:32.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:32.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:32.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:33.625 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:38:33.617+0000 7fa033e1f7c0 -1 osd.5 0 log_to_monitors true 2024-09-16T16:38:33.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:33.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:33.727 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:38:33.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:38:33.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:33.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:34.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:35.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:35.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:35.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:38:35.255 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:38:35.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:35.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:35.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:36.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:36.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:36.717 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:38:36.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:38:36.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:36.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:37.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:38.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:38.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:38.180 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:38:38.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:38:38.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:38.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:38.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:39.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:39.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:39.729 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:38:39.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:38:39.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:39.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:40.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:41.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:41.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:41.212 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:38:41.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:38:41.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:41.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:41.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:42.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:42.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:42.721 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:38:42.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:38:42.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:42.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:43.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:44.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:44.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:44.164 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:38:44.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:38:44.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:44.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:44.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:45.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:45.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:45.711 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:38:45.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:38:45.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:45.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:46.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:47.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:47.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:47.216 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:38:47.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:38:47.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:47.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:48.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:49.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:49.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:49.389 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:38:49.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:38:49.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:49.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:50.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:51.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:51.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:51.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:38:51.225 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:38:51.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:51.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:52.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:53.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:53.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:53.033 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:38:53.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:38:53.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:53.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:53.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:54.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:54.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:54.677 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:38:54.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:38:54.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:54.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:55.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:56.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:56.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:56.774 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:38:56.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:38:56.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:56.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:57.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:38:58.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:38:58.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:38:58.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:38:58.583 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:38:58.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:38:58.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:38:59.013 INFO:tasks.workunit.client.0.smithi039.stdout:osd.5 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3415802158,v1:127.0.0.1:6843/3415802158] [v2:127.0.0.1:6844/3415802158,v1:127.0.0.1:6845/3415802158] exists,up 870d170c-df6e-4670-bb43-1861ba08b2f4 2024-09-16T16:38:59.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:38:59.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:38:59.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:38:59.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:451: TEST_recovery_multi: create_pool test 1 1 2024-09-16T16:38:59.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-16T16:38:59.744 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T16:38:59.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:39:00.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:452: TEST_recovery_multi: ceph osd pool set test size 3 2024-09-16T16:39:01.619 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 3 2024-09-16T16:39:01.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:453: TEST_recovery_multi: ceph osd pool set test min_size 1 2024-09-16T16:39:02.651 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 1 2024-09-16T16:39:02.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:455: TEST_recovery_multi: wait_for_clean 2024-09-16T16:39:02.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:39:02.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:39:02.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:39:02.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:39:02.680 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:39:02.680 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:39:02.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:39:02.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:39:02.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:39:02.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:39:02.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:39:02.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:39:02.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:39:02.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:39:02.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:39:03.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:39:03.254 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:39:03.254 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:39:03.254 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:39:03.254 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T16:39:03.254 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T16:39:03.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:39:03.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:39:03.254 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:39:03.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836522 2024-09-16T16:39:03.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836522 2024-09-16T16:39:03.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836522' 2024-09-16T16:39:03.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:39:03.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:39:03.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=38654705702 2024-09-16T16:39:03.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 38654705702 2024-09-16T16:39:03.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836522 1-38654705702' 2024-09-16T16:39:03.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:39:03.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:39:03.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=60129542171 2024-09-16T16:39:03.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 60129542171 2024-09-16T16:39:03.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836522 1-38654705702 2-60129542171' 2024-09-16T16:39:03.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:39:03.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:39:03.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378648 2024-09-16T16:39:03.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378648 2024-09-16T16:39:03.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836522 1-38654705702 2-60129542171 3-81604378648' 2024-09-16T16:39:03.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:39:03.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:39:04.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=98784247823 2024-09-16T16:39:04.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 98784247823 2024-09-16T16:39:04.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836522 1-38654705702 2-60129542171 3-81604378648 4-98784247823' 2024-09-16T16:39:04.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:39:04.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T16:39:04.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084297 2024-09-16T16:39:04.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084297 2024-09-16T16:39:04.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836522 1-38654705702 2-60129542171 3-81604378648 4-98784247823 5-120259084297' 2024-09-16T16:39:04.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:39:04.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836522 2024-09-16T16:39:04.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:39:04.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:39:04.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836522 2024-09-16T16:39:04.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:39:04.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836522 2024-09-16T16:39:04.264 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836522 2024-09-16T16:39:04.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-09-16T16:39:04.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:39:04.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836522 -lt 21474836522 2024-09-16T16:39:04.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:39:04.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-38654705702 2024-09-16T16:39:04.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:39:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:39:04.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-38654705702 2024-09-16T16:39:04.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:39:04.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=38654705702 2024-09-16T16:39:04.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 38654705702' 2024-09-16T16:39:04.681 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 38654705702 2024-09-16T16:39:04.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:39:05.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 38654705700 -lt 38654705702 2024-09-16T16:39:05.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:39:06.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:39:06.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:39:06.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 38654705702 -lt 38654705702 2024-09-16T16:39:06.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:39:06.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-60129542171 2024-09-16T16:39:06.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:39:06.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:39:06.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-60129542171 2024-09-16T16:39:06.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:39:06.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=60129542171 2024-09-16T16:39:06.565 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 60129542171 2024-09-16T16:39:06.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 60129542171' 2024-09-16T16:39:06.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:39:07.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 60129542171 -lt 60129542171 2024-09-16T16:39:07.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:39:07.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:39:07.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-81604378648 2024-09-16T16:39:07.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:39:07.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-81604378648 2024-09-16T16:39:07.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:39:07.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378648 2024-09-16T16:39:07.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 81604378648' 2024-09-16T16:39:07.030 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 81604378648 2024-09-16T16:39:07.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:39:07.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378648 -lt 81604378648 2024-09-16T16:39:07.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:39:07.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:39:07.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-98784247823 2024-09-16T16:39:07.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:39:07.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-98784247823 2024-09-16T16:39:07.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:39:07.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=98784247823 2024-09-16T16:39:07.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 98784247823' 2024-09-16T16:39:07.453 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 98784247823 2024-09-16T16:39:07.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:39:07.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 98784247823 -lt 98784247823 2024-09-16T16:39:07.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:39:07.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-120259084297 2024-09-16T16:39:07.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:39:07.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T16:39:07.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-120259084297 2024-09-16T16:39:07.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:39:07.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084297 2024-09-16T16:39:07.913 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 120259084297 2024-09-16T16:39:07.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 120259084297' 2024-09-16T16:39:07.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T16:39:08.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084297 -lt 120259084297 2024-09-16T16:39:08.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:39:08.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:39:08.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:39:08.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:39:08.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:39:08.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:39:08.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:39:08.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:39:08.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:39:08.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:39:08.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:39:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:39:09.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:39:09.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:39:09.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:39:09.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:39:09.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:39:09.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:39:09.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:457: TEST_recovery_multi: rados -p test put obj1 /dev/null 2024-09-16T16:39:10.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: get_primary test obj1 2024-09-16T16:39:10.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:39:10.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:39:10.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:39:10.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:39:10.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: local primary=1 2024-09-16T16:39:10.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: get_not_primary test obj1 2024-09-16T16:39:10.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-16T16:39:10.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-16T16:39:10.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-16T16:39:10.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:39:10.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:39:10.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:39:10.442 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:39:10.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-16T16:39:10.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-16T16:39:10.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-16T16:39:11.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: local otherosd=0 2024-09-16T16:39:11.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:462: TEST_recovery_multi: ceph osd set noout 2024-09-16T16:39:12.255 INFO:tasks.workunit.client.0.smithi039.stderr:noout is set 2024-09-16T16:39:12.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:463: TEST_recovery_multi: ceph osd set norecover 2024-09-16T16:39:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is set 2024-09-16T16:39:13.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: cat td/osd-recovery-stats/osd.0.pid 2024-09-16T16:39:13.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: kill 305190 2024-09-16T16:39:13.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:465: TEST_recovery_multi: ceph osd down osd.0 2024-09-16T16:39:13.620 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T16:39:13.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: expr 200 / 2 2024-09-16T16:39:13.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: local half=100 2024-09-16T16:39:13.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: seq 2 100 2024-09-16T16:39:13.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:13.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj2 /dev/null 2024-09-16T16:39:14.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:14.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj3 /dev/null 2024-09-16T16:39:14.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:14.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj4 /dev/null 2024-09-16T16:39:14.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:14.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj5 /dev/null 2024-09-16T16:39:14.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:14.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj6 /dev/null 2024-09-16T16:39:14.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:14.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj7 /dev/null 2024-09-16T16:39:15.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:15.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj8 /dev/null 2024-09-16T16:39:15.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:15.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj9 /dev/null 2024-09-16T16:39:15.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:15.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj10 /dev/null 2024-09-16T16:39:15.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:15.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj11 /dev/null 2024-09-16T16:39:15.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:15.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj12 /dev/null 2024-09-16T16:39:15.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:15.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj13 /dev/null 2024-09-16T16:39:15.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:15.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj14 /dev/null 2024-09-16T16:39:16.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:16.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj15 /dev/null 2024-09-16T16:39:16.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:16.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj16 /dev/null 2024-09-16T16:39:16.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:16.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj17 /dev/null 2024-09-16T16:39:16.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:16.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj18 /dev/null 2024-09-16T16:39:16.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:16.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj19 /dev/null 2024-09-16T16:39:16.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:16.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj20 /dev/null 2024-09-16T16:39:16.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:16.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj21 /dev/null 2024-09-16T16:39:16.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:16.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj22 /dev/null 2024-09-16T16:39:17.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:17.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj23 /dev/null 2024-09-16T16:39:17.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:17.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj24 /dev/null 2024-09-16T16:39:17.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:17.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj25 /dev/null 2024-09-16T16:39:17.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:17.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj26 /dev/null 2024-09-16T16:39:17.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:17.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj27 /dev/null 2024-09-16T16:39:17.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:17.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj28 /dev/null 2024-09-16T16:39:17.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:17.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj29 /dev/null 2024-09-16T16:39:18.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:18.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj30 /dev/null 2024-09-16T16:39:18.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:18.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj31 /dev/null 2024-09-16T16:39:18.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:18.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj32 /dev/null 2024-09-16T16:39:18.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:18.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj33 /dev/null 2024-09-16T16:39:18.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:18.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj34 /dev/null 2024-09-16T16:39:18.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:18.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj35 /dev/null 2024-09-16T16:39:18.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:18.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj36 /dev/null 2024-09-16T16:39:19.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:19.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj37 /dev/null 2024-09-16T16:39:19.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:19.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj38 /dev/null 2024-09-16T16:39:19.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:19.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj39 /dev/null 2024-09-16T16:39:19.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:19.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj40 /dev/null 2024-09-16T16:39:19.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:19.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj41 /dev/null 2024-09-16T16:39:19.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:19.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj42 /dev/null 2024-09-16T16:39:19.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:19.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj43 /dev/null 2024-09-16T16:39:19.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:19.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj44 /dev/null 2024-09-16T16:39:20.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:20.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj45 /dev/null 2024-09-16T16:39:20.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:20.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj46 /dev/null 2024-09-16T16:39:20.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:20.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj47 /dev/null 2024-09-16T16:39:20.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:20.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj48 /dev/null 2024-09-16T16:39:20.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:20.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj49 /dev/null 2024-09-16T16:39:20.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:20.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj50 /dev/null 2024-09-16T16:39:20.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:20.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj51 /dev/null 2024-09-16T16:39:21.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:21.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj52 /dev/null 2024-09-16T16:39:21.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:21.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj53 /dev/null 2024-09-16T16:39:21.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:21.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj54 /dev/null 2024-09-16T16:39:21.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:21.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj55 /dev/null 2024-09-16T16:39:21.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:21.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj56 /dev/null 2024-09-16T16:39:21.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:21.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj57 /dev/null 2024-09-16T16:39:21.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:21.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj58 /dev/null 2024-09-16T16:39:21.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:21.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj59 /dev/null 2024-09-16T16:39:21.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:21.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj60 /dev/null 2024-09-16T16:39:22.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:22.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj61 /dev/null 2024-09-16T16:39:22.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:22.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj62 /dev/null 2024-09-16T16:39:22.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:22.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj63 /dev/null 2024-09-16T16:39:22.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:22.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj64 /dev/null 2024-09-16T16:39:22.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:22.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj65 /dev/null 2024-09-16T16:39:22.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:22.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj66 /dev/null 2024-09-16T16:39:22.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:22.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj67 /dev/null 2024-09-16T16:39:22.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:22.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj68 /dev/null 2024-09-16T16:39:23.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:23.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj69 /dev/null 2024-09-16T16:39:23.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:23.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj70 /dev/null 2024-09-16T16:39:23.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:23.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj71 /dev/null 2024-09-16T16:39:23.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:23.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj72 /dev/null 2024-09-16T16:39:23.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:23.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj73 /dev/null 2024-09-16T16:39:23.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:23.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj74 /dev/null 2024-09-16T16:39:23.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:23.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj75 /dev/null 2024-09-16T16:39:24.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:24.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj76 /dev/null 2024-09-16T16:39:24.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:24.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj77 /dev/null 2024-09-16T16:39:24.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:24.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj78 /dev/null 2024-09-16T16:39:24.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:24.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj79 /dev/null 2024-09-16T16:39:24.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:24.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj80 /dev/null 2024-09-16T16:39:24.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:24.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj81 /dev/null 2024-09-16T16:39:24.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:24.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj82 /dev/null 2024-09-16T16:39:24.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:24.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj83 /dev/null 2024-09-16T16:39:25.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:25.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj84 /dev/null 2024-09-16T16:39:25.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:25.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj85 /dev/null 2024-09-16T16:39:25.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:25.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj86 /dev/null 2024-09-16T16:39:25.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:25.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj87 /dev/null 2024-09-16T16:39:25.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:25.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj88 /dev/null 2024-09-16T16:39:25.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:25.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj89 /dev/null 2024-09-16T16:39:25.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:25.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj90 /dev/null 2024-09-16T16:39:25.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:25.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj91 /dev/null 2024-09-16T16:39:26.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:26.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj92 /dev/null 2024-09-16T16:39:26.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:26.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj93 /dev/null 2024-09-16T16:39:26.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:26.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj94 /dev/null 2024-09-16T16:39:26.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:26.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj95 /dev/null 2024-09-16T16:39:26.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:26.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj96 /dev/null 2024-09-16T16:39:26.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:26.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj97 /dev/null 2024-09-16T16:39:26.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:26.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj98 /dev/null 2024-09-16T16:39:26.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:26.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj99 /dev/null 2024-09-16T16:39:27.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-16T16:39:27.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj100 /dev/null 2024-09-16T16:39:27.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: cat td/osd-recovery-stats/osd.1.pid 2024-09-16T16:39:27.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: kill 306903 2024-09-16T16:39:27.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:474: TEST_recovery_multi: ceph osd down osd.1 2024-09-16T16:39:27.795 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-16T16:39:27.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:475: TEST_recovery_multi: activate_osd td/osd-recovery-stats 0 2024-09-16T16:39:27.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-09-16T16:39:27.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T16:39:27.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T16:39:27.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T16:39:27.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/0 2024-09-16T16:39:27.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:39:27.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:39:27.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:39:27.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:39:27.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-16T16:39:27.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-16T16:39:27.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T16:39:27.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T16:39:27.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:39:27.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T16:39:27.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:39:27.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:39:27.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:39:27.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:39:27.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:39:27.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:39:27.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:39:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:39:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:39:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:39:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:39:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:39:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:39:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:39:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T16:39:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T16:39:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/0 2024-09-16T16:39:27.826 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:39:27.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T16:39:27.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/0/whoami 2024-09-16T16:39:27.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:39:27.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T16:39:27.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T16:39:27.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T16:39:27.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T16:39:27.851 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:39:27.848+0000 7fe4c9c9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:39:27.980 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:39:27.976+0000 7fe4c9c9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:39:27.982 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:39:27.980+0000 7fe4c9c9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:39:28.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T16:39:28.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:39:28.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:39:28.288 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:39:28.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:39:28.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:39:28.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:39:28.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:39:28.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:39:28.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:39:28.512 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:39:28.508+0000 7fe4c9c9d7c0 -1 Falling back to public interface 2024-09-16T16:39:28.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:39:29.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:39:29.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:39:29.730 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:39:29.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:39:29.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:39:29.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:39:30.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:39:31.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:39:31.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:39:31.163 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:39:31.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:39:31.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:39:31.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:39:31.418 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:39:31.416+0000 7fe4c9c9d7c0 -1 osd.0 38 log_to_monitors true 2024-09-16T16:39:31.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:39:32.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:39:32.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:39:32.679 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:39:32.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:39:32.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:39:32.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:39:33.256 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 45 up_thru 45 down_at 40 last_clean_interval [5,38) [v2:127.0.0.1:6802/1646621939,v1:127.0.0.1:6803/1646621939] [v2:127.0.0.1:6804/1646621939,v1:127.0.0.1:6805/1646621939] exists,up 8a474966-f593-4873-9315-ee6384b1da9e 2024-09-16T16:39:33.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:39:33.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:39:33.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:39:33.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:476: TEST_recovery_multi: sleep 3 2024-09-16T16:39:36.260 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: expr 100 + 1 2024-09-16T16:39:36.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: seq 101 200 2024-09-16T16:39:36.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:36.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj101 /dev/null 2024-09-16T16:39:36.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:36.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj102 /dev/null 2024-09-16T16:39:36.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:36.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj103 /dev/null 2024-09-16T16:39:36.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:36.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj104 /dev/null 2024-09-16T16:39:36.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:36.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj105 /dev/null 2024-09-16T16:39:36.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:36.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj106 /dev/null 2024-09-16T16:39:37.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:37.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj107 /dev/null 2024-09-16T16:39:37.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:37.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj108 /dev/null 2024-09-16T16:39:37.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:37.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj109 /dev/null 2024-09-16T16:39:37.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:37.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj110 /dev/null 2024-09-16T16:39:37.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:37.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj111 /dev/null 2024-09-16T16:39:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj112 /dev/null 2024-09-16T16:39:37.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:37.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj113 /dev/null 2024-09-16T16:39:38.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:38.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj114 /dev/null 2024-09-16T16:39:38.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:38.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj115 /dev/null 2024-09-16T16:39:38.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:38.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj116 /dev/null 2024-09-16T16:39:38.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:38.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj117 /dev/null 2024-09-16T16:39:38.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:38.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj118 /dev/null 2024-09-16T16:39:38.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:38.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj119 /dev/null 2024-09-16T16:39:38.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:38.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj120 /dev/null 2024-09-16T16:39:38.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:38.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj121 /dev/null 2024-09-16T16:39:39.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:39.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj122 /dev/null 2024-09-16T16:39:39.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:39.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj123 /dev/null 2024-09-16T16:39:39.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:39.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj124 /dev/null 2024-09-16T16:39:39.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:39.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj125 /dev/null 2024-09-16T16:39:39.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:39.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj126 /dev/null 2024-09-16T16:39:39.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:39.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj127 /dev/null 2024-09-16T16:39:39.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:39.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj128 /dev/null 2024-09-16T16:39:39.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:39.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj129 /dev/null 2024-09-16T16:39:40.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:40.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj130 /dev/null 2024-09-16T16:39:40.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:40.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj131 /dev/null 2024-09-16T16:39:40.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:40.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj132 /dev/null 2024-09-16T16:39:40.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:40.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj133 /dev/null 2024-09-16T16:39:40.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:40.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj134 /dev/null 2024-09-16T16:39:40.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:40.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj135 /dev/null 2024-09-16T16:39:40.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:40.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj136 /dev/null 2024-09-16T16:39:40.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:40.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj137 /dev/null 2024-09-16T16:39:41.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:41.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj138 /dev/null 2024-09-16T16:39:41.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:41.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj139 /dev/null 2024-09-16T16:39:41.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:41.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj140 /dev/null 2024-09-16T16:39:41.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:41.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj141 /dev/null 2024-09-16T16:39:41.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:41.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj142 /dev/null 2024-09-16T16:39:41.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:41.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj143 /dev/null 2024-09-16T16:39:41.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:41.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj144 /dev/null 2024-09-16T16:39:41.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:41.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj145 /dev/null 2024-09-16T16:39:42.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:42.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj146 /dev/null 2024-09-16T16:39:42.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:42.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj147 /dev/null 2024-09-16T16:39:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:42.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj148 /dev/null 2024-09-16T16:39:42.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:42.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj149 /dev/null 2024-09-16T16:39:42.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:42.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj150 /dev/null 2024-09-16T16:39:42.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:42.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj151 /dev/null 2024-09-16T16:39:42.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:42.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj152 /dev/null 2024-09-16T16:39:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj153 /dev/null 2024-09-16T16:39:43.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:43.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj154 /dev/null 2024-09-16T16:39:43.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:43.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj155 /dev/null 2024-09-16T16:39:43.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:43.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj156 /dev/null 2024-09-16T16:39:43.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:43.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj157 /dev/null 2024-09-16T16:39:43.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:43.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj158 /dev/null 2024-09-16T16:39:43.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:43.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj159 /dev/null 2024-09-16T16:39:43.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:43.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj160 /dev/null 2024-09-16T16:39:44.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:44.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj161 /dev/null 2024-09-16T16:39:44.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:44.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj162 /dev/null 2024-09-16T16:39:44.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:44.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj163 /dev/null 2024-09-16T16:39:44.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:44.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj164 /dev/null 2024-09-16T16:39:44.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:44.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj165 /dev/null 2024-09-16T16:39:44.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:44.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj166 /dev/null 2024-09-16T16:39:44.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:44.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj167 /dev/null 2024-09-16T16:39:44.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:44.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj168 /dev/null 2024-09-16T16:39:45.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:45.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj169 /dev/null 2024-09-16T16:39:45.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:45.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj170 /dev/null 2024-09-16T16:39:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj171 /dev/null 2024-09-16T16:39:45.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:45.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj172 /dev/null 2024-09-16T16:39:45.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:45.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj173 /dev/null 2024-09-16T16:39:45.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:45.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj174 /dev/null 2024-09-16T16:39:45.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:45.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj175 /dev/null 2024-09-16T16:39:45.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:45.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj176 /dev/null 2024-09-16T16:39:46.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:46.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj177 /dev/null 2024-09-16T16:39:46.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:46.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj178 /dev/null 2024-09-16T16:39:46.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:46.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj179 /dev/null 2024-09-16T16:39:46.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:46.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj180 /dev/null 2024-09-16T16:39:46.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:46.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj181 /dev/null 2024-09-16T16:39:46.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:46.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj182 /dev/null 2024-09-16T16:39:46.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:46.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj183 /dev/null 2024-09-16T16:39:46.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:46.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj184 /dev/null 2024-09-16T16:39:47.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:47.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj185 /dev/null 2024-09-16T16:39:47.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:47.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj186 /dev/null 2024-09-16T16:39:47.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:47.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj187 /dev/null 2024-09-16T16:39:47.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:47.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj188 /dev/null 2024-09-16T16:39:47.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:47.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj189 /dev/null 2024-09-16T16:39:47.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:47.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj190 /dev/null 2024-09-16T16:39:47.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:47.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj191 /dev/null 2024-09-16T16:39:48.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:48.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj192 /dev/null 2024-09-16T16:39:48.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:48.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj193 /dev/null 2024-09-16T16:39:48.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:48.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj194 /dev/null 2024-09-16T16:39:48.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:48.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj195 /dev/null 2024-09-16T16:39:48.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:48.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj196 /dev/null 2024-09-16T16:39:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:48.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj197 /dev/null 2024-09-16T16:39:48.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:48.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj198 /dev/null 2024-09-16T16:39:49.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:49.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj199 /dev/null 2024-09-16T16:39:49.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-16T16:39:49.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj200 /dev/null 2024-09-16T16:39:49.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: get_pg test obj1 2024-09-16T16:39:49.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-16T16:39:49.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-16T16:39:49.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-16T16:39:49.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T16:39:49.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: local PG=1.0 2024-09-16T16:39:49.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: get_not_primary test obj200 2024-09-16T16:39:49.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-16T16:39:49.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj200 2024-09-16T16:39:49.703 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj200 2024-09-16T16:39:49.703 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:39:49.703 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj200 2024-09-16T16:39:49.703 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj200 2024-09-16T16:39:49.703 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:39:50.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=0 2024-09-16T16:39:50.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj200 2024-09-16T16:39:50.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 0)) | .[0]' 2024-09-16T16:39:50.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: local otherosd=2 2024-09-16T16:39:50.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:486: TEST_recovery_multi: ceph osd unset noout 2024-09-16T16:39:51.469 INFO:tasks.workunit.client.0.smithi039.stderr:noout is unset 2024-09-16T16:39:51.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:487: TEST_recovery_multi: ceph osd out osd.1 osd.2 2024-09-16T16:39:52.192 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already out. osd.2 is already out. 2024-09-16T16:39:52.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:488: TEST_recovery_multi: activate_osd td/osd-recovery-stats 1 2024-09-16T16:39:52.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-09-16T16:39:52.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T16:39:52.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T16:39:52.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T16:39:52.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/1 2024-09-16T16:39:52.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:39:52.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:39:52.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:39:52.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:39:52.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-16T16:39:52.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-16T16:39:52.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T16:39:52.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T16:39:52.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:39:52.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T16:39:52.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:39:52.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:39:52.212 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:39:52.212 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:39:52.212 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:39:52.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:39:52.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:39:52.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:39:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:39:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:39:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:39:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:39:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:39:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:39:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T16:39:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T16:39:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/1 2024-09-16T16:39:52.214 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:39:52.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T16:39:52.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:39:52.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/1/whoami 2024-09-16T16:39:52.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T16:39:52.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T16:39:52.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T16:39:52.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T16:39:52.246 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:39:52.243+0000 7f23704b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:39:52.279 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:39:52.275+0000 7f23704b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:39:52.280 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:39:52.275+0000 7f23704b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:39:52.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T16:39:52.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:39:52.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:39:52.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:39:52.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:39:52.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:39:52.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:39:52.655 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:39:52.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:39:52.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:39:52.828 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:39:52.823+0000 7f23704b07c0 -1 Falling back to public interface 2024-09-16T16:39:53.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:39:54.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:39:54.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:39:54.222 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:39:54.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:39:54.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:39:54.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:39:54.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:39:55.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:39:55.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:39:55.639 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:39:55.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:39:55.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:39:55.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:39:55.874 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:39:55.867+0000 7f23704b07c0 -1 osd.1 41 log_to_monitors true 2024-09-16T16:39:56.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:39:57.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:39:57.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:39:57.081 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:39:57.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:39:57.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:39:57.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:39:57.507 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up out weight 0 up_from 54 up_thru 40 down_at 42 last_clean_interval [9,41) [v2:127.0.0.1:6810/1090303191,v1:127.0.0.1:6811/1090303191] [v2:127.0.0.1:6812/1090303191,v1:127.0.0.1:6813/1090303191] exists,up e8b8a2b4-8e30-4c8c-8e29-496baf8755bf 2024-09-16T16:39:57.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:39:57.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:39:57.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:39:57.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:489: TEST_recovery_multi: sleep 3 2024-09-16T16:40:00.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:491: TEST_recovery_multi: ceph osd pool set test size 4 2024-09-16T16:40:01.419 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 4 2024-09-16T16:40:01.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:492: TEST_recovery_multi: ceph osd unset norecover 2024-09-16T16:40:03.156 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is unset 2024-09-16T16:40:03.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: get_primary test obj1 2024-09-16T16:40:03.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:40:03.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:40:03.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:40:03.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:40:03.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-16T16:40:03.889 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T16:40:03.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:494: TEST_recovery_multi: sleep 2 2024-09-16T16:40:05.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:496: TEST_recovery_multi: wait_for_clean 2024-09-16T16:40:05.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:40:05.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:40:05.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:40:05.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:40:05.909 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:40:05.909 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:40:05.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:40:05.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:40:05.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:40:06.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:40:06.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:40:06.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:40:06.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:40:06.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:40:06.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:40:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:40:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:40:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:40:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:40:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T16:40:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T16:40:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:40:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:40:06.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:40:06.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=193273528329 2024-09-16T16:40:06.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 193273528329 2024-09-16T16:40:06.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-193273528329' 2024-09-16T16:40:06.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:40:06.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:40:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=231928233988 2024-09-16T16:40:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 231928233988 2024-09-16T16:40:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-193273528329 1-231928233988' 2024-09-16T16:40:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:40:06.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:40:06.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=60129542185 2024-09-16T16:40:06.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 60129542185 2024-09-16T16:40:06.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-193273528329 1-231928233988 2-60129542185' 2024-09-16T16:40:06.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:40:06.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:40:07.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378662 2024-09-16T16:40:07.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378662 2024-09-16T16:40:07.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-193273528329 1-231928233988 2-60129542185 3-81604378662' 2024-09-16T16:40:07.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:40:07.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:40:07.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=98784247837 2024-09-16T16:40:07.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 98784247837 2024-09-16T16:40:07.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-193273528329 1-231928233988 2-60129542185 3-81604378662 4-98784247837' 2024-09-16T16:40:07.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:40:07.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T16:40:07.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084311 2024-09-16T16:40:07.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084311 2024-09-16T16:40:07.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-193273528329 1-231928233988 2-60129542185 3-81604378662 4-98784247837 5-120259084311' 2024-09-16T16:40:07.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:40:07.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-193273528329 2024-09-16T16:40:07.457 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:40:07.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:40:07.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-193273528329 2024-09-16T16:40:07.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:40:07.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=193273528329 2024-09-16T16:40:07.463 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 193273528329 2024-09-16T16:40:07.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 193273528329' 2024-09-16T16:40:07.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:40:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 193273528330 -lt 193273528329 2024-09-16T16:40:07.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:40:07.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-231928233988 2024-09-16T16:40:07.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:40:07.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:40:07.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-231928233988 2024-09-16T16:40:07.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:40:07.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=231928233988 2024-09-16T16:40:07.966 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 231928233988 2024-09-16T16:40:07.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 231928233988' 2024-09-16T16:40:07.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:40:08.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 231928233989 -lt 231928233988 2024-09-16T16:40:08.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:40:08.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:40:08.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-60129542185 2024-09-16T16:40:08.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:40:08.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-60129542185 2024-09-16T16:40:08.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:40:08.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=60129542185 2024-09-16T16:40:08.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 60129542185' 2024-09-16T16:40:08.407 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 60129542185 2024-09-16T16:40:08.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:40:08.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 60129542186 -lt 60129542185 2024-09-16T16:40:08.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:40:08.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-81604378662 2024-09-16T16:40:08.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:40:08.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:40:08.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-81604378662 2024-09-16T16:40:08.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:40:08.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378662 2024-09-16T16:40:08.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 81604378662' 2024-09-16T16:40:08.928 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 81604378662 2024-09-16T16:40:08.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:40:09.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378663 -lt 81604378662 2024-09-16T16:40:09.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:40:09.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-98784247837 2024-09-16T16:40:09.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:40:09.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:40:09.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-98784247837 2024-09-16T16:40:09.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:40:09.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=98784247837 2024-09-16T16:40:09.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 98784247837' 2024-09-16T16:40:09.450 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 98784247837 2024-09-16T16:40:09.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:40:09.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 98784247838 -lt 98784247837 2024-09-16T16:40:09.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:40:09.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-120259084311 2024-09-16T16:40:09.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:40:09.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T16:40:09.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-120259084311 2024-09-16T16:40:09.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:40:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084311 2024-09-16T16:40:09.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 120259084311' 2024-09-16T16:40:09.981 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 120259084311 2024-09-16T16:40:09.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T16:40:10.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084312 -lt 120259084311 2024-09-16T16:40:10.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:40:10.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:10.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:11.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:40:11.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:11.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:11.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:11.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:11.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:11.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:11.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:11.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:11.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:11.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:11.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:12.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:12.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T16:40:12.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T16:40:12.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T16:40:12.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:12.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:12.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:12.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:12.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:12.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:12.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:12.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:12.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:12.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:12.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:12.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:12.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:12.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:13.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:13.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:13.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:13.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:13.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:13.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:13.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:13.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:13.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-16T16:40:14.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-16T16:40:14.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:14.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:14.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:14.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:14.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:14.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:14.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:14.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:14.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:14.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:14.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:14.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:14.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:14.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:14.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:15.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:15.206 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:15.206 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:15.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-09-16T16:40:15.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-09-16T16:40:15.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:15.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:15.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:15.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:15.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:15.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:15.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:15.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:15.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:15.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:15.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:16.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:16.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:16.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:16.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:17.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:17.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:17.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:17.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:17.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-09-16T16:40:17.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-09-16T16:40:17.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:17.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:17.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:17.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:17.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:17.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:17.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:17.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:17.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:17.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:17.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:18.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:18.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:18.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:18.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:18.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:18.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:18.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:18.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:18.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:18.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:18.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:18.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:18.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:19.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-16T16:40:19.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-16T16:40:19.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:19.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:19.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:19.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:19.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:19.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:19.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:19.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:19.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:19.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:19.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:20.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:20.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:20.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:20.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:20.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:20.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:21.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-16T16:40:21.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-16T16:40:21.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:21.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:21.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:21.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:21.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:21.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:21.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:21.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:21.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:21.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:21.625 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:22.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:22.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:22.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:22.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:22.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:22.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:22.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:22.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:22.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:22.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:22.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:22.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:22.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:23.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-16T16:40:23.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-16T16:40:23.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:23.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:23.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:23.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:23.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:23.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:23.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:23.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:23.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:23.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:23.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:23.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:23.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:23.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:23.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:24.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:24.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:24.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:24.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:24.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:24.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:24.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:24.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:24.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:25.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-09-16T16:40:25.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-09-16T16:40:25.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:25.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:25.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:25.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:25.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:25.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:25.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:25.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:25.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:25.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:25.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:25.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:25.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:25.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:25.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:26.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:26.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:26.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:26.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-16T16:40:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-16T16:40:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:27.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:27.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:27.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:27.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:27.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:27.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:27.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:27.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:27.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:27.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:27.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:27.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:27.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:28.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:28.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:28.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:28.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:28.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:28.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:28.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:28.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:28.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:28.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-09-16T16:40:28.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-09-16T16:40:28.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:28.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:28.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:29.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:29.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:29.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:29.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:29.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:29.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:29.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:29.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:29.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:29.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:29.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:29.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:30.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:30.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:30.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:30.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:30.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:30.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:30.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:30.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:30.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:30.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-16T16:40:30.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-16T16:40:30.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:30.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:30.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:30.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:30.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:30.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:30.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:30.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:30.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:30.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:30.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:31.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:31.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:31.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:31.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:31.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:31.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:31.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:31.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:31.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:31.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:31.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:31.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:31.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:32.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-16T16:40:32.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-16T16:40:32.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:32.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:32.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:32.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:32.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:32.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:32.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:32.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:32.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:32.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:32.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:32.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:32.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:32.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:32.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:33.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:33.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:33.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:33.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:33.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:33.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:33.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:33.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:33.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:34.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-16T16:40:34.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-16T16:40:34.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:34.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:34.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:34.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:34.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:34.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:34.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:34.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:34.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:34.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:34.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:34.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:34.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:34.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:34.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:35.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:35.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:35.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:35.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:35.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:35.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:35.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:35.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:35.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:36.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-16T16:40:36.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-16T16:40:36.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:36.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:36.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:36.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:36.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:36.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:36.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:36.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:36.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:36.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:36.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:36.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:40:36.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:36.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:36.759 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:37.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:40:37.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:40:37.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:40:37.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:40:37.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:40:37.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:40:37.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:40:37.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:40:37.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:40:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-16T16:40:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-16T16:40:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:40:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:40:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:40:37.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:40:37.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:40:37.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:40:37.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:40:37.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:40:37.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:40:37.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:40:37.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:40:38.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:40:38.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:40:38.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:40:38.409 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:40:38.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:40:38.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:40:38.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:40:38.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: get_primary test obj1 2024-09-16T16:40:38.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:40:38.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:40:38.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:40:38.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:40:39.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: primary=0 2024-09-16T16:40:39.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:501: TEST_recovery_multi: local log=td/osd-recovery-stats/osd.0.log 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:502: TEST_recovery_multi: check td/osd-recovery-stats 1.0 0 replicated 399 0 300 0 99 0 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=0 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=399 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=300 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=99 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.0.log 2024-09-16T16:40:39.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-16T16:40:39.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-16T16:40:39.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-16T16:40:39.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.0.log 2024-09-16T16:40:39.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-16T16:40:39.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-16T16:40:39.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[0,4,3,5]/[0,3]' 2024-09-16T16:40:39.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-16T16:40:39.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:40:39.424 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-09-16T16:40:39.424 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-09-16T16:40:39.424 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-16T16:40:39.425 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-16T16:40:39.425 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:40:39.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=399 2024-09-16T16:40:39.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 399 399 2024-09-16T16:40:39.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=399 2024-09-16T16:40:39.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:40:39.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=399 2024-09-16T16:40:39.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:40:39.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:40:39.431 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-09-16T16:40:39.432 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-16T16:40:39.432 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:40:39.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-16T16:40:39.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-16T16:40:39.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:40:39.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:40:39.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:40:39.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:40:39.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-16T16:40:39.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:40:39.445 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-09-16T16:40:39.445 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-09-16T16:40:39.445 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-16T16:40:39.445 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-16T16:40:39.446 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:40:39.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=300 2024-09-16T16:40:39.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 300 300 2024-09-16T16:40:39.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=300 2024-09-16T16:40:39.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:40:39.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=300 2024-09-16T16:40:39.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:40:39.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:40:39.451 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-09-16T16:40:39.451 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-16T16:40:39.451 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:40:39.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-16T16:40:39.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-16T16:40:39.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:40:39.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:40:39.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:40:39.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:40:39.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 99 ']' 2024-09-16T16:40:39.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 0' 2024-09-16T16:40:39.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:40:39.463 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-09-16T16:40:39.463 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-09-16T16:40:39.463 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-09-16T16:40:39.463 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-09-16T16:40:39.463 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:40:39.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=99 2024-09-16T16:40:39.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 99 99 2024-09-16T16:40:39.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=99 2024-09-16T16:40:39.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:40:39.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=99 2024-09-16T16:40:39.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:40:39.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:40:39.469 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-09-16T16:40:39.469 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-09-16T16:40:39.469 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:40:39.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-09-16T16:40:39.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-09-16T16:40:39.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:40:39.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:40:39.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:40:39.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:40:39.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:504: TEST_recovery_multi: delete_pool test 2024-09-16T16:40:39.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T16:40:39.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T16:40:40.183 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T16:40:40.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:505: TEST_recovery_multi: kill_daemons td/osd-recovery-stats 2024-09-16T16:40:40.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:40:40.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:40:40.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:40:40.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:40:40.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:40:45.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:40:45.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-16T16:40:45.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:40:45.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:40:45.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:40:45.522 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:40:45.522 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:40:45.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:40:45.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:40:45.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:40:45.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:40:45.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:40:45.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:40:45.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:40:45.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:40:45.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:40:45.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:40:45.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:40:45.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:40:45.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:40:45.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:40:45.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:40:45.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:40:45.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:40:45.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:40:45.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:40:45.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:40:45.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:40:45.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:40:45.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:40:45.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-16T16:40:45.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-16T16:40:45.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-16T16:40:45.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-16T16:40:45.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:40:45.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:40:45.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:40:45.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:40:45.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:40:45.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:40:45.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:40:45.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:40:45.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:40:45.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:40:45.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:40:45.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:40:45.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:40:45.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:40:45.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:40:45.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:40:45.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:40:45.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:40:45.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:40:45.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:40:45.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:40:45.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:40:45.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:40:45.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:40:45.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:40:45.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:40:45.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:40:45.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:40:45.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-16T16:40:45.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:40:45.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:40:45.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:40:45.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274941 2024-09-16T16:40:45.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:40:45.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T16:40:45.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:40:45.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-16T16:40:45.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_replicated_out1 td/osd-recovery-stats 2024-09-16T16:40:45.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:167: TEST_recovery_replicated_out1: local dir=td/osd-recovery-stats 2024-09-16T16:40:45.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:169: TEST_recovery_replicated_out1: do_recovery_out1 td/osd-recovery-stats replicated 2024-09-16T16:40:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-09-16T16:40:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-09-16T16:40:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=replicated 2024-09-16T16:40:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-09-16T16:40:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-16T16:40:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:40:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:40:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:40:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-16T16:40:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-16T16:40:45.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:40:45.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:40:45.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:40:45.960 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:40:45.960 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:40:45.960 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:40:45.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:40:45.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:40:46.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:40:46.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:40:46.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:40:46.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:40:46.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:40:46.189 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:40:46.189 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:40:46.189 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:40:46.189 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:40:46.189 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:40:46.189 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:40:46.189 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:40:46.190 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:40:46.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:40:46.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get fsid 2024-09-16T16:40:46.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:40:46.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:40:46.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:40:46.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:40:46.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:40:46.309 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:40:46.309 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:40:46.309 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:40:46.309 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:40:46.309 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:40:46.310 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:40:46.310 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:40:46.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:40:46.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get mon_host 2024-09-16T16:40:46.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-09-16T16:40:46.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-16T16:40:46.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:40:46.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:40:46.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:40:46.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-16T16:40:46.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:40:47.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:40:47.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:40:47.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:40:47.349 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:40:47.349 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:40:47.349 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:40:47.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:40:47.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:40:47.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:40:47.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-09-16T16:40:47.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:40:47.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:40:47.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:40:47.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:40:47.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-16T16:40:47.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:40:47.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:40:47.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:40:47.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:40:47.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:40:47.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-16T16:40:47.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:40:47.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=61baa50c-9202-444f-92d9-fa5455af36da 2024-09-16T16:40:47.424 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 61baa50c-9202-444f-92d9-fa5455af36da 2024-09-16T16:40:47.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 61baa50c-9202-444f-92d9-fa5455af36da' 2024-09-16T16:40:47.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:40:47.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPX+hmu4tOGhAAoPVWjoD76k34lE5FO6f+6w== 2024-09-16T16:40:47.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPX+hmu4tOGhAAoPVWjoD76k34lE5FO6f+6w=="}' 2024-09-16T16:40:47.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 61baa50c-9202-444f-92d9-fa5455af36da -i td/osd-recovery-stats/0/new.json 2024-09-16T16:40:47.802 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:40:47.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-16T16:40:47.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCPX+hmu4tOGhAAoPVWjoD76k34lE5FO6f+6w== --osd-uuid 61baa50c-9202-444f-92d9-fa5455af36da 2024-09-16T16:40:47.846 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:47.846+0000 7f9a15d3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:40:47.849 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:47.846+0000 7f9a15d3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:40:47.850 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:47.850+0000 7f9a15d3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:40:47.850 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:47.850+0000 7f9a15d3a7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:40:47.850 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:47.850+0000 7f9a15d3a7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:40:47.851 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:47.850+0000 7f9a15d3a7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:40:47.934 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:47.930+0000 7f9a15d3a7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-16T16:40:54.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-16T16:40:54.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:40:54.081 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:40:54.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:40:54.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:40:54.647 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:40:54.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:40:54.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:40:54.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:40:54.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:40:54.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:40:54.680 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:54.678+0000 7fac6e9187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:40:54.682 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:54.682+0000 7fac6e9187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:40:54.684 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:54.682+0000 7fac6e9187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:40:55.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:40:55.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:40:55.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:40:55.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:40:55.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:40:55.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:40:55.076 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:40:55.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:40:55.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:40:55.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:40:55.255 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:55.254+0000 7fac6e9187c0 -1 Falling back to public interface 2024-09-16T16:40:55.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:40:56.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:40:56.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:40:56.498 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:40:56.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:40:56.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:40:56.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:40:56.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:40:57.857 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:40:57.854+0000 7fac6e9187c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:40:57.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:40:57.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:40:57.914 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:40:57.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:40:57.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:40:57.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:40:58.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:40:59.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:40:59.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:40:59.427 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:40:59.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:40:59.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:40:59.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:40:59.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:00.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:00.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:00.880 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:41:00.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:41:00.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:00.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:01.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:02.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:02.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:02.338 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:41:02.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:41:02.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:02.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:02.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:03.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:03.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:03.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:41:03.841 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:41:03.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:03.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:04.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:05.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:05.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:05.325 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:41:05.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:41:05.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:05.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:05.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:06.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:06.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:06.873 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:41:06.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:41:06.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:06.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:07.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:08.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:08.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:08.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:41:08.327 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:41:08.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:08.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:08.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:09.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:09.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:09.859 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:41:09.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:41:09.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:09.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:10.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:11.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:11.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:11.339 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:41:11.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:41:11.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:11.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:11.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:12.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:12.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:12.812 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:41:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:41:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:12.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:13.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:14.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:14.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:14.550 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:41:14.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:41:14.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:14.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:15.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:16.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:16.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:16.286 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:41:16.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:41:16.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:16.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:16.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:17.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:17.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:17.789 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:41:17.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:41:17.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:17.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:18.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:19.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:19.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:19.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:41:19.603 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:41:19.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:19.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:20.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:21.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:21.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:21.331 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:41:21.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:41:21.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:21.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:41:21.794 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3703414123,v1:127.0.0.1:6803/3703414123] [v2:127.0.0.1:6804/3703414123,v1:127.0.0.1:6805/3703414123] exists,up 61baa50c-9202-444f-92d9-fa5455af36da 2024-09-16T16:41:21.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:41:21.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:41:21.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:41:21.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:41:21.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:41:21.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:41:21.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:41:21.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:41:21.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:41:21.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:41:21.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:41:21.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:41:21.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:41:21.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-16T16:41:21.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:41:21.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe80f941-e6ed-45a8-bb53-b5070358deb2 2024-09-16T16:41:21.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fe80f941-e6ed-45a8-bb53-b5070358deb2' 2024-09-16T16:41:21.802 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 fe80f941-e6ed-45a8-bb53-b5070358deb2 2024-09-16T16:41:21.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:41:21.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxX+hm23++MBAAR4CWihIHA+tmbHLh8bbyAA== 2024-09-16T16:41:21.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxX+hm23++MBAAR4CWihIHA+tmbHLh8bbyAA=="}' 2024-09-16T16:41:21.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe80f941-e6ed-45a8-bb53-b5070358deb2 -i td/osd-recovery-stats/1/new.json 2024-09-16T16:41:22.431 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:41:22.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-16T16:41:22.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxX+hm23++MBAAR4CWihIHA+tmbHLh8bbyAA== --osd-uuid fe80f941-e6ed-45a8-bb53-b5070358deb2 2024-09-16T16:41:22.479 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:22.477+0000 7f5119ee37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:41:22.481 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:22.477+0000 7f5119ee37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:41:22.482 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:22.481+0000 7f5119ee37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:41:22.482 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:22.481+0000 7f5119ee37c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:41:22.482 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:22.481+0000 7f5119ee37c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:41:22.483 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:22.481+0000 7f5119ee37c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:41:22.611 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:22.609+0000 7f5119ee37c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-16T16:41:28.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-16T16:41:28.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:41:28.403 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:41:28.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:41:28.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:41:29.028 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:41:29.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:41:29.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:41:29.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:41:29.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:41:29.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:41:29.062 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:29.061+0000 7f7e265af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:41:29.065 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:29.061+0000 7f7e265af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:41:29.066 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:29.065+0000 7f7e265af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:41:29.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:41:29.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:41:29.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:41:29.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:41:29.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:41:29.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:29.542 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:41:29.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:41:29.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:29.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:29.667 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:29.665+0000 7f7e265af7c0 -1 Falling back to public interface 2024-09-16T16:41:29.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:30.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:30.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:30.972 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:41:30.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:41:30.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:30.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:31.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:32.185 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:32.181+0000 7f7e265af7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:41:32.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:32.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:32.391 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:41:32.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:41:32.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:32.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:32.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:33.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:33.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:33.847 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:41:33.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:41:33.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:33.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:34.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:35.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:35.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:35.340 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:41:35.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:41:35.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:35.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:35.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:36.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:36.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:36.760 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:41:36.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:41:36.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:36.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:37.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:38.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:38.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:38.260 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:41:38.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:41:38.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:38.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:38.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:39.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:39.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:39.762 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:41:39.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:41:39.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:39.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:40.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:41.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:41.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:41.310 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:41:41.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:41:41.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:41.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:41.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:42.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:42.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:42.763 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:41:42.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:41:42.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:42.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:43.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:44.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:44.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:44.374 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:41:44.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:41:44.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:44.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:44.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:45.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:45.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:45.911 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:41:45.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:41:45.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:45.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:46.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:47.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:47.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:47.710 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:41:47.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:41:47.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:47.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:48.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:49.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:49.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:49.627 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:41:49.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:41:49.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:49.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:50.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:51.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:51.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:51.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:41:51.394 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:41:51.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:51.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:52.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:53.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:53.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:53.378 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:41:53.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:41:53.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:53.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:54.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:55.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:55.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:55.148 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:41:55.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:41:55.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:55.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:56.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:41:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:41:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:41:57.078 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:41:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:41:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:41:57.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:41:57.533 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2706361269,v1:127.0.0.1:6811/2706361269] [v2:127.0.0.1:6812/2706361269,v1:127.0.0.1:6813/2706361269] exists,up fe80f941-e6ed-45a8-bb53-b5070358deb2 2024-09-16T16:41:57.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:41:57.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:41:57.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:41:57.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-09-16T16:41:57.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:41:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:41:57.538 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:41:57.538 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:41:57.538 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:41:57.538 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:41:57.538 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:41:57.538 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:41:57.538 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:41:57.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:41:57.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:41:57.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:41:57.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:41:57.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:41:57.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:41:57.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:41:57.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:41:57.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:41:57.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:41:57.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:41:57.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:41:57.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-16T16:41:57.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:41:57.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2e74551-6501-4306-867e-10e2e8ee8932 2024-09-16T16:41:57.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b2e74551-6501-4306-867e-10e2e8ee8932' 2024-09-16T16:41:57.540 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 b2e74551-6501-4306-867e-10e2e8ee8932 2024-09-16T16:41:57.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:41:57.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVX+hm2SSFIRAASgxLGyRMCvFp91yc40nyAg== 2024-09-16T16:41:57.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVX+hm2SSFIRAASgxLGyRMCvFp91yc40nyAg=="}' 2024-09-16T16:41:57.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2e74551-6501-4306-867e-10e2e8ee8932 -i td/osd-recovery-stats/2/new.json 2024-09-16T16:41:58.155 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:41:58.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-16T16:41:58.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVX+hm2SSFIRAASgxLGyRMCvFp91yc40nyAg== --osd-uuid b2e74551-6501-4306-867e-10e2e8ee8932 2024-09-16T16:41:58.212 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:58.209+0000 7fb0e94687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:41:58.245 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:58.241+0000 7fb0e94687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:41:58.247 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:58.245+0000 7fb0e94687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:41:58.247 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:58.245+0000 7fb0e94687c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:41:58.247 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:58.245+0000 7fb0e94687c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:41:58.248 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:58.245+0000 7fb0e94687c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:41:58.370 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:41:58.365+0000 7fb0e94687c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-16T16:42:04.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-16T16:42:04.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:42:04.429 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T16:42:04.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T16:42:04.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:42:05.032 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T16:42:05.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T16:42:05.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:42:05.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:42:05.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:42:05.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:42:05.065 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:05.061+0000 7fc7a5c4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:42:05.068 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:05.065+0000 7fc7a5c4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:42:05.069 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:05.065+0000 7fc7a5c4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:42:05.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T16:42:05.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:42:05.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T16:42:05.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:42:05.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:42:05.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:05.440 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:42:05.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:42:05.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:05.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:05.624 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:05.621+0000 7fc7a5c4d7c0 -1 Falling back to public interface 2024-09-16T16:42:05.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:06.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:06.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:06.862 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:42:06.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:42:06.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:06.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:07.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:08.210 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:08.204+0000 7fc7a5c4d7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T16:42:08.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:08.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:08.327 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:42:08.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:42:08.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:08.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:08.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:09.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:09.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:09.806 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:42:09.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:42:09.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:09.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:10.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:11.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:11.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:11.290 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:42:11.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:42:11.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:11.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:11.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:12.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:12.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:12.728 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:42:12.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:42:12.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:12.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:13.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:14.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:14.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:14.260 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:42:14.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:42:14.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:14.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:14.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:15.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:15.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:15.831 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:42:15.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:42:15.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:15.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:16.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:17.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:17.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:17.269 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:42:17.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:42:17.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:17.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:17.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:18.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:18.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:18.724 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:42:18.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:42:18.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:18.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:19.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:20.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:20.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:20.273 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:42:20.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:42:20.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:20.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:20.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:21.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:21.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:21.773 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:42:21.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:42:21.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:21.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:22.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:23.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:23.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:23.604 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:42:23.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:42:23.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:23.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:24.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:25.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:25.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:25.449 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:42:25.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:42:25.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:25.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:26.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:27.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:27.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:27.159 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:42:27.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:42:27.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:27.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:28.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:29.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:29.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:29.023 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:42:29.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:42:29.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:29.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:30.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:31.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:31.028 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:42:31.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:31.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:42:31.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:31.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:31.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:32.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:32.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:32.526 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:42:32.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:42:32.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:32.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:42:32.974 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/757358371,v1:127.0.0.1:6819/757358371] [v2:127.0.0.1:6820/757358371,v1:127.0.0.1:6821/757358371] exists,up b2e74551-6501-4306-867e-10e2e8ee8932 2024-09-16T16:42:32.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:42:32.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:42:32.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:42:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:42:32.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:42:32.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:42:32.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:42:32.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:42:32.979 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:42:32.979 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:42:32.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:42:32.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:42:32.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:42:32.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-16T16:42:32.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:42:32.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c496268b-0b94-4dae-b446-a30da8c81b35 2024-09-16T16:42:32.982 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 c496268b-0b94-4dae-b446-a30da8c81b35 2024-09-16T16:42:32.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 c496268b-0b94-4dae-b446-a30da8c81b35' 2024-09-16T16:42:32.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:42:33.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5X+hm5ToyABAA+b3PqEaQflm+8hHwciqgTg== 2024-09-16T16:42:33.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5X+hm5ToyABAA+b3PqEaQflm+8hHwciqgTg=="}' 2024-09-16T16:42:33.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c496268b-0b94-4dae-b446-a30da8c81b35 -i td/osd-recovery-stats/3/new.json 2024-09-16T16:42:33.559 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:42:33.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-16T16:42:33.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5X+hm5ToyABAA+b3PqEaQflm+8hHwciqgTg== --osd-uuid c496268b-0b94-4dae-b446-a30da8c81b35 2024-09-16T16:42:33.616 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:33.612+0000 7f2be770f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:42:33.639 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:33.636+0000 7f2be770f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:42:33.641 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:33.636+0000 7f2be770f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:42:33.642 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:33.636+0000 7f2be770f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:42:33.642 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:33.636+0000 7f2be770f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:42:33.642 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:33.636+0000 7f2be770f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:42:33.731 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:33.728+0000 7f2be770f7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-16T16:42:39.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-16T16:42:39.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:42:39.885 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T16:42:39.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T16:42:39.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:42:40.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T16:42:40.449 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T16:42:40.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:42:40.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:42:40.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:42:40.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:42:40.474 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:40.472+0000 7f53074ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:42:40.497 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:40.492+0000 7f53074ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:42:40.498 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:40.496+0000 7f53074ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:42:40.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T16:42:40.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:42:40.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T16:42:40.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:42:40.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:42:40.865 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:42:40.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:40.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:42:40.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:40.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:41.052 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:41.048+0000 7f53074ee7c0 -1 Falling back to public interface 2024-09-16T16:42:41.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:42.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:42.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:42.304 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:42:42.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:42:42.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:42.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:42.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:43.619 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:42:43.616+0000 7f53074ee7c0 -1 osd.3 0 log_to_monitors true 2024-09-16T16:42:43.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:43.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:43.727 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:42:43.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:42:43.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:43.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:44.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:45.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:45.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:45.340 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:42:45.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:42:45.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:45.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:45.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:46.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:46.769 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:42:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:42:46.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:46.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:47.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:48.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:48.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:48.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:42:48.403 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:42:48.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:48.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:48.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:49.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:49.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:49.957 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:42:49.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:42:49.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:49.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:50.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:51.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:51.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:51.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:42:51.613 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:42:51.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:51.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:52.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:53.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:53.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:53.046 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:42:53.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:42:53.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:53.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:53.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:54.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:54.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:54.542 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:42:54.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:42:54.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:54.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:54.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:55.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:55.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:55.996 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:42:55.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:42:55.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:55.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:57.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:58.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:58.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:58.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:42:58.019 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:42:58.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:58.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:42:58.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:42:59.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:42:59.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:42:59.605 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:42:59.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:42:59.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:42:59.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:43:00.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:01.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:01.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:01.660 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:43:01.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:43:01.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:01.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:43:02.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:03.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:03.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:03.488 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:43:03.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:43:03.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:03.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:43:04.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:05.229 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:43:05.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:43:05.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:05.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:43:05.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:06.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:06.912 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:43:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:43:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:06.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:43:07.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:08.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:08.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:08.422 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:43:08.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:43:08.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:08.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:43:08.866 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1851711059,v1:127.0.0.1:6827/1851711059] [v2:127.0.0.1:6828/1851711059,v1:127.0.0.1:6829/1851711059] exists,up c496268b-0b94-4dae-b446-a30da8c81b35 2024-09-16T16:43:08.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:43:08.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:43:08.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-16T16:43:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:43:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:43:08.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:43:08.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:43:08.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:43:08.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:43:08.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:43:08.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:43:08.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-16T16:43:08.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:43:08.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1303c3d4-0daa-40a3-96f7-ecda26264991 2024-09-16T16:43:08.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 1303c3d4-0daa-40a3-96f7-ecda26264991' 2024-09-16T16:43:08.873 INFO:tasks.workunit.client.0.smithi039.stdout:add osd4 1303c3d4-0daa-40a3-96f7-ecda26264991 2024-09-16T16:43:08.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:43:08.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcYOhm0Nk7NRAAM9042qatUY1y4VJl/bXlaw== 2024-09-16T16:43:08.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcYOhm0Nk7NRAAM9042qatUY1y4VJl/bXlaw=="}' 2024-09-16T16:43:08.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1303c3d4-0daa-40a3-96f7-ecda26264991 -i td/osd-recovery-stats/4/new.json 2024-09-16T16:43:09.499 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:43:09.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-16T16:43:09.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAcYOhm0Nk7NRAAM9042qatUY1y4VJl/bXlaw== --osd-uuid 1303c3d4-0daa-40a3-96f7-ecda26264991 2024-09-16T16:43:09.544 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:09.539+0000 7f183f0ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:09.546 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:09.543+0000 7f183f0ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:09.547 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:09.543+0000 7f183f0ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:09.548 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:09.543+0000 7f183f0ed7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:43:09.548 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:09.543+0000 7f183f0ed7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:43:09.548 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:09.543+0000 7f183f0ed7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:43:09.637 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:09.631+0000 7f183f0ed7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-16T16:43:15.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-16T16:43:15.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:43:15.861 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd4 key to auth repository 2024-09-16T16:43:15.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T16:43:15.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:43:16.449 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.4 2024-09-16T16:43:16.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T16:43:16.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:43:16.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:43:16.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:43:16.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:43:16.482 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:16.479+0000 7f1847af27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:16.503 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:16.499+0000 7f1847af27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:16.504 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:16.499+0000 7f1847af27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:16.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T16:43:16.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:43:16.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T16:43:16.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:43:16.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:43:16.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:16.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:43:16.890 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:43:16.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:16.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:17.095 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:17.091+0000 7f1847af27c0 -1 Falling back to public interface 2024-09-16T16:43:17.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:18.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:18.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:18.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:43:18.313 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:43:18.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:18.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:18.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:19.713 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:19.707+0000 7f1847af27c0 -1 osd.4 0 log_to_monitors true 2024-09-16T16:43:19.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:19.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:19.761 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:43:19.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:43:19.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:19.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:20.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:21.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:21.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:21.539 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:43:21.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:43:21.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:21.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:21.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:22.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:22.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:22.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:43:22.984 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:43:22.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:22.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:23.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:24.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:24.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:24.514 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:43:24.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:43:24.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:24.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:25.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:26.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:26.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:26.115 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:43:26.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:43:26.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:26.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:26.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:27.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:27.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:27.831 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:43:27.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:43:27.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:27.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:28.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:29.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:29.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:29.285 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:43:29.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:43:29.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:29.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:29.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:30.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:30.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:30.853 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:43:30.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:43:30.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:30.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:31.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:32.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:32.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:32.375 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:43:32.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:43:32.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:32.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:32.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:33.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:33.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:33.958 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:43:33.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:43:33.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:33.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:34.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:35.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:35.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:35.481 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:43:35.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:43:35.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:35.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:36.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:37.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:37.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:37.452 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:43:37.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:43:37.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:37.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:38.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:39.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:39.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:39.346 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:43:39.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:43:39.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:39.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:40.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:41.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:41.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:41.063 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:43:41.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:43:41.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:41.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:42.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:43.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:43.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:43.195 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:43:43.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:43:43.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:43.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:43:43.708 INFO:tasks.workunit.client.0.smithi039.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1950925094,v1:127.0.0.1:6835/1950925094] [v2:127.0.0.1:6836/1950925094,v1:127.0.0.1:6837/1950925094] exists,up 1303c3d4-0daa-40a3-96f7-ecda26264991 2024-09-16T16:43:43.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:43:43.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:43:43.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:43:43.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-09-16T16:43:43.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:43:43.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:43:43.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T16:43:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:43:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-16T16:43:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:43:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:43:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:43:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:43:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-16T16:43:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-16T16:43:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:43:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:43:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:43:43.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:43:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:43:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:43:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:43:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:43:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:43:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-16T16:43:43.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:43:43.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=97275eb7-0747-432a-b1d1-0e8bca849c51 2024-09-16T16:43:43.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 97275eb7-0747-432a-b1d1-0e8bca849c51' 2024-09-16T16:43:43.716 INFO:tasks.workunit.client.0.smithi039.stdout:add osd5 97275eb7-0747-432a-b1d1-0e8bca849c51 2024-09-16T16:43:43.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:43:43.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/YOhmr6USLBAA/kze35kv1S+XEyL3I1LzHQ== 2024-09-16T16:43:43.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/YOhmr6USLBAA/kze35kv1S+XEyL3I1LzHQ=="}' 2024-09-16T16:43:43.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 97275eb7-0747-432a-b1d1-0e8bca849c51 -i td/osd-recovery-stats/5/new.json 2024-09-16T16:43:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:43:44.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-16T16:43:44.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/YOhmr6USLBAA/kze35kv1S+XEyL3I1LzHQ== --osd-uuid 97275eb7-0747-432a-b1d1-0e8bca849c51 2024-09-16T16:43:44.433 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:44.431+0000 7f1312e757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:44.450 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:44.451+0000 7f1312e757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:44.452 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:44.451+0000 7f1312e757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:44.452 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:44.451+0000 7f1312e757c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:43:44.452 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:44.451+0000 7f1312e757c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:43:44.453 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:44.451+0000 7f1312e757c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:43:44.585 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:44.583+0000 7f1312e757c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-16T16:43:50.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-16T16:43:50.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:43:50.690 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd5 key to auth repository 2024-09-16T16:43:50.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T16:43:50.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:43:51.255 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.5 2024-09-16T16:43:51.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T16:43:51.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:43:51.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:43:51.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:43:51.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:43:51.289 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:51.286+0000 7fc8306af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:51.310 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:51.310+0000 7fc8306af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:51.312 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:51.310+0000 7fc8306af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:43:51.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T16:43:51.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:43:51.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T16:43:51.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:43:51.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:43:51.716 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:43:51.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:51.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:43:51.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:51.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:43:51.892 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:51.890+0000 7fc8306af7c0 -1 Falling back to public interface 2024-09-16T16:43:52.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:53.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:53.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:53.148 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:43:53.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:43:53.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:53.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:43:53.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:54.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:43:54.582+0000 7fc8306af7c0 -1 osd.5 0 log_to_monitors true 2024-09-16T16:43:54.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:54.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:54.621 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:43:54.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:43:54.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:54.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:43:55.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:56.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:56.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:56.162 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:43:56.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:43:56.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:56.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:43:56.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:57.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:57.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:57.713 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:43:57.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:43:57.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:57.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:43:58.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:43:59.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:43:59.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:43:59.163 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:43:59.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:43:59.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:43:59.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:43:59.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:00.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:00.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:00.595 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:44:00.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:44:00.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:00.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:01.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:02.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:02.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:02.146 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:44:02.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:44:02.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:02.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:03.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:03.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:03.659 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:44:03.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:44:03.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:03.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:04.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:05.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:05.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:05.120 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:44:05.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:44:05.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:05.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:05.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:06.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:06.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:06.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:44:06.611 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:44:06.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:06.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:07.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:08.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:08.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:08.178 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:44:08.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:44:08.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:08.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:09.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:10.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:10.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:10.290 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:44:10.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:44:10.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:10.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:11.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:12.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:12.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:12.077 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:44:12.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:44:12.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:12.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:12.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:13.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:13.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:13.564 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:44:13.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:44:13.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:13.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:14.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:15.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:15.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:15.276 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:44:15.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:44:15.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:15.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:15.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:16.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:16.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:16.870 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:44:16.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:44:16.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:16.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:17.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:18.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:18.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:18.532 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:44:18.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:44:18.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:18.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:18.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:44:19.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:44:19.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:44:19.986 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T16:44:19.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T16:44:19.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:44:19.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:44:20.411 INFO:tasks.workunit.client.0.smithi039.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1622092223,v1:127.0.0.1:6843/1622092223] [v2:127.0.0.1:6844/1622092223,v1:127.0.0.1:6845/1622092223] exists,up 97275eb7-0747-432a-b1d1-0e8bca849c51 2024-09-16T16:44:20.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:44:20.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:44:20.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:44:20.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' replicated = erasure ']' 2024-09-16T16:44:20.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:135: do_recovery_out1: create_pool test 1 1 replicated 2024-09-16T16:44:20.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 replicated 2024-09-16T16:44:20.997 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T16:44:21.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:44:22.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-09-16T16:44:22.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:44:22.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:44:22.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:44:22.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:44:22.017 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:44:22.017 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:44:22.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:44:22.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:44:22.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:44:22.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:44:22.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:44:22.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:44:22.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:44:22.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:44:22.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:44:22.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:44:22.653 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:44:22.653 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:44:22.653 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:44:22.653 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T16:44:22.653 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T16:44:22.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:44:22.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:44:22.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:44:22.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836523 2024-09-16T16:44:22.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836523 2024-09-16T16:44:22.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523' 2024-09-16T16:44:22.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:44:22.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:44:23.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672997 2024-09-16T16:44:23.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672997 2024-09-16T16:44:23.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949672997' 2024-09-16T16:44:23.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:44:23.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:44:23.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509468 2024-09-16T16:44:23.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509468 2024-09-16T16:44:23.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949672997 2-64424509468' 2024-09-16T16:44:23.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:44:23.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:44:23.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345939 2024-09-16T16:44:23.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345939 2024-09-16T16:44:23.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949672997 2-64424509468 3-85899345939' 2024-09-16T16:44:23.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:44:23.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:44:23.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182416 2024-09-16T16:44:23.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182416 2024-09-16T16:44:23.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949672997 2-64424509468 3-85899345939 4-107374182416' 2024-09-16T16:44:23.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:44:23.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T16:44:23.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018888 2024-09-16T16:44:23.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018888 2024-09-16T16:44:23.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949672997 2-64424509468 3-85899345939 4-107374182416 5-128849018888' 2024-09-16T16:44:23.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:44:23.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836523 2024-09-16T16:44:23.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:44:23.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:44:23.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836523 2024-09-16T16:44:23.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:44:23.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836523 2024-09-16T16:44:23.689 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836523 2024-09-16T16:44:23.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-09-16T16:44:23.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:44:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836521 -lt 21474836523 2024-09-16T16:44:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:44:25.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:44:25.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:44:25.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836523 -lt 21474836523 2024-09-16T16:44:25.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:44:25.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672997 2024-09-16T16:44:25.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:44:25.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:44:25.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672997 2024-09-16T16:44:25.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:44:25.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672997 2024-09-16T16:44:25.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2024-09-16T16:44:25.575 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672997 2024-09-16T16:44:25.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:44:26.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672997 -lt 42949672997 2024-09-16T16:44:26.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:44:26.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509468 2024-09-16T16:44:26.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:44:26.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:44:26.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509468 2024-09-16T16:44:26.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:44:26.005 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509468 2024-09-16T16:44:26.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509468 2024-09-16T16:44:26.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-09-16T16:44:26.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:44:26.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509468 -lt 64424509468 2024-09-16T16:44:26.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:44:26.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345939 2024-09-16T16:44:26.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:44:26.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:44:26.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:44:26.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345939 2024-09-16T16:44:26.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345939 2024-09-16T16:44:26.547 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345939 2024-09-16T16:44:26.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2024-09-16T16:44:26.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:44:26.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345939 -lt 85899345939 2024-09-16T16:44:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:44:26.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182416 2024-09-16T16:44:26.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:44:26.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:44:26.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182416 2024-09-16T16:44:26.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:44:26.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182416 2024-09-16T16:44:26.977 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182416 2024-09-16T16:44:26.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182416' 2024-09-16T16:44:26.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:44:27.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182417 -lt 107374182416 2024-09-16T16:44:27.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:44:27.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018888 2024-09-16T16:44:27.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:44:27.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T16:44:27.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018888 2024-09-16T16:44:27.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:44:27.405 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018888 2024-09-16T16:44:27.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018888 2024-09-16T16:44:27.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2024-09-16T16:44:27.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T16:44:27.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018889 -lt 128849018888 2024-09-16T16:44:27.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:44:27.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:44:27.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:44:28.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:44:28.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:44:28.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:44:28.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:44:28.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:44:28.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:44:28.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:44:28.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:44:28.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:44:28.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:44:28.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:44:28.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:44:29.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:44:29.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:44:29.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:44:29.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-09-16T16:44:29.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:29.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-09-16T16:44:29.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:29.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-09-16T16:44:29.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:29.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-09-16T16:44:29.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:29.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-09-16T16:44:29.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:29.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-09-16T16:44:30.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:30.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-09-16T16:44:30.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:30.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-09-16T16:44:30.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:30.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-09-16T16:44:30.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:30.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-09-16T16:44:30.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:30.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-09-16T16:44:30.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:30.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-09-16T16:44:30.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:30.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-09-16T16:44:31.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:31.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-09-16T16:44:31.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:31.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-09-16T16:44:31.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:31.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-09-16T16:44:31.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:31.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-09-16T16:44:31.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:31.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-09-16T16:44:31.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:31.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-09-16T16:44:31.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:31.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-09-16T16:44:31.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:31.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-09-16T16:44:32.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:32.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-09-16T16:44:32.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:32.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-09-16T16:44:32.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:32.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-09-16T16:44:32.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:32.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-09-16T16:44:32.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:32.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-09-16T16:44:32.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:32.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-09-16T16:44:32.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:32.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-09-16T16:44:32.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:32.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-09-16T16:44:33.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:33.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-09-16T16:44:33.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:33.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-09-16T16:44:33.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:33.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-09-16T16:44:33.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:33.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-09-16T16:44:33.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:33.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-09-16T16:44:33.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:33.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-09-16T16:44:33.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:33.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-09-16T16:44:34.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:34.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-09-16T16:44:34.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:34.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-09-16T16:44:34.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:34.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-09-16T16:44:34.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:34.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-09-16T16:44:34.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:34.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-09-16T16:44:34.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:34.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-09-16T16:44:34.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:34.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-09-16T16:44:34.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:34.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-09-16T16:44:35.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:35.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-09-16T16:44:35.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:35.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-09-16T16:44:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:35.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-09-16T16:44:35.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:35.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-09-16T16:44:35.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:35.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-09-16T16:44:35.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:35.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-09-16T16:44:35.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:35.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-09-16T16:44:35.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:35.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-09-16T16:44:36.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:36.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-09-16T16:44:36.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:36.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-09-16T16:44:36.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:36.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-09-16T16:44:36.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:36.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-09-16T16:44:36.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:36.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-09-16T16:44:36.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:36.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-09-16T16:44:36.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:36.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-09-16T16:44:37.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:37.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-09-16T16:44:37.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:37.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-09-16T16:44:37.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:37.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-09-16T16:44:37.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:37.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-09-16T16:44:37.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:37.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-09-16T16:44:37.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:37.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-09-16T16:44:37.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:37.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-09-16T16:44:37.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:37.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-09-16T16:44:38.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:38.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-09-16T16:44:38.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:38.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-09-16T16:44:38.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:38.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-09-16T16:44:38.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:38.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-09-16T16:44:38.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:38.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-09-16T16:44:38.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:38.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-09-16T16:44:38.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:38.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-09-16T16:44:39.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:39.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-09-16T16:44:39.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:39.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-09-16T16:44:39.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:39.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-09-16T16:44:39.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:39.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-09-16T16:44:39.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:39.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-09-16T16:44:39.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:39.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-09-16T16:44:39.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:39.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-09-16T16:44:39.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:39.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-09-16T16:44:40.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:40.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-09-16T16:44:40.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:40.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-09-16T16:44:40.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:40.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-09-16T16:44:40.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:40.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-09-16T16:44:40.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:40.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-09-16T16:44:40.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:40.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-09-16T16:44:40.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:40.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-09-16T16:44:40.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:40.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-09-16T16:44:41.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:41.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-09-16T16:44:41.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:41.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-09-16T16:44:41.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:41.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-09-16T16:44:41.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:41.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-09-16T16:44:41.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:41.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-09-16T16:44:41.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:41.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-09-16T16:44:41.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:41.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-09-16T16:44:41.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:41.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-09-16T16:44:42.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:42.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-09-16T16:44:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:42.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-09-16T16:44:42.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:42.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-09-16T16:44:42.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:42.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-09-16T16:44:42.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:42.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-09-16T16:44:42.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:42.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-09-16T16:44:42.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:42.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-09-16T16:44:42.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:42.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-09-16T16:44:43.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:43.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-09-16T16:44:43.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:43.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-09-16T16:44:43.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:43.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-09-16T16:44:43.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:43.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-09-16T16:44:43.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:43.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-09-16T16:44:43.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:43.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-09-16T16:44:43.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:43.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-09-16T16:44:44.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:44.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-09-16T16:44:44.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:44.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-09-16T16:44:44.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:44.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-09-16T16:44:44.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:44.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-09-16T16:44:44.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:44.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-09-16T16:44:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-09-16T16:44:44.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:44.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-09-16T16:44:44.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:44.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-09-16T16:44:45.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:45.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-09-16T16:44:45.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:45.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-09-16T16:44:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:45.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-09-16T16:44:45.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:45.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-09-16T16:44:45.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:45.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-09-16T16:44:45.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:45.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-09-16T16:44:45.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:45.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-09-16T16:44:45.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:45.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-09-16T16:44:45.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:45.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-09-16T16:44:46.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:46.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-09-16T16:44:46.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:46.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-09-16T16:44:46.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:46.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-09-16T16:44:46.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:46.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-09-16T16:44:46.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:46.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-09-16T16:44:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-09-16T16:44:46.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:46.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-09-16T16:44:47.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:47.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-09-16T16:44:47.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:47.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-09-16T16:44:47.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:47.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-09-16T16:44:47.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:47.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-09-16T16:44:47.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:47.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-09-16T16:44:47.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:47.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-09-16T16:44:47.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:47.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-09-16T16:44:47.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:47.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-09-16T16:44:48.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:48.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-09-16T16:44:48.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:48.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-09-16T16:44:48.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:48.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-09-16T16:44:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-09-16T16:44:48.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:48.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-09-16T16:44:48.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:48.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-09-16T16:44:48.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:48.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-09-16T16:44:48.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:48.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-09-16T16:44:49.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:49.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-09-16T16:44:49.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:49.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-09-16T16:44:49.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:49.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-09-16T16:44:49.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:49.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-09-16T16:44:49.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:49.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-09-16T16:44:49.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:49.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-09-16T16:44:49.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:49.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-09-16T16:44:49.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:49.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-09-16T16:44:50.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:50.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-09-16T16:44:50.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:50.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-09-16T16:44:50.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:50.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-09-16T16:44:50.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:50.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-09-16T16:44:50.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:50.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-09-16T16:44:50.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:50.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-09-16T16:44:50.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:50.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-09-16T16:44:51.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:51.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-09-16T16:44:51.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:51.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-09-16T16:44:51.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:51.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-09-16T16:44:51.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:51.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-09-16T16:44:51.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:51.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-09-16T16:44:51.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:51.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-09-16T16:44:51.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:51.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-09-16T16:44:51.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:51.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-09-16T16:44:52.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:52.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-09-16T16:44:52.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:52.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-09-16T16:44:52.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:52.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-09-16T16:44:52.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:52.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-09-16T16:44:52.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:52.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-09-16T16:44:52.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:52.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-09-16T16:44:52.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:52.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-09-16T16:44:52.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:52.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-09-16T16:44:53.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:53.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-09-16T16:44:53.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:53.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-09-16T16:44:53.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:53.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-09-16T16:44:53.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:53.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-09-16T16:44:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:53.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-09-16T16:44:53.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:53.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-09-16T16:44:53.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:53.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-09-16T16:44:54.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:54.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-09-16T16:44:54.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:54.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-09-16T16:44:54.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:54.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-09-16T16:44:54.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:54.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-09-16T16:44:54.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:54.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-09-16T16:44:54.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:54.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-09-16T16:44:54.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:54.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-09-16T16:44:54.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:54.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-09-16T16:44:54.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:54.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-09-16T16:44:55.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-16T16:44:55.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-09-16T16:44:55.257 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-09-16T16:44:55.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:44:55.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:44:55.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:44:55.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:44:55.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-09-16T16:44:55.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-09-16T16:44:55.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-16T16:44:55.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-16T16:44:55.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-16T16:44:55.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T16:44:56.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-09-16T16:44:56.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-09-16T16:44:56.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-16T16:44:56.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-16T16:44:56.120 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-16T16:44:56.120 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:44:56.120 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:44:56.120 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:44:56.120 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:44:56.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-16T16:44:56.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-16T16:44:56.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-16T16:44:57.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-09-16T16:44:57.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-09-16T16:44:57.948 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is set 2024-09-16T16:44:57.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-09-16T16:44:57.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 323674 2024-09-16T16:44:57.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-09-16T16:44:58.823 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T16:44:58.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-09-16T16:44:59.434 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T16:44:59.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-09-16T16:45:00.178 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is unset 2024-09-16T16:45:00.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-09-16T16:45:00.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:45:00.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:45:00.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:45:00.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:45:00.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-16T16:45:01.097 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T16:45:01.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-09-16T16:45:03.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-09-16T16:45:03.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:45:03.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:45:03.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:45:03.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:45:03.119 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:45:03.119 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:45:03.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:45:03.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:45:03.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:45:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:45:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:45:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:45:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:45:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:45:03.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:45:03.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:45:03.682 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:45:03.682 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:45:03.683 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:45:03.683 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T16:45:03.683 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T16:45:03.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:45:03.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:45:03.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:45:03.853 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-16T16:45:03.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T16:45:03.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T16:45:03.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T16:45:03.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:45:03.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:45:04.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673007 2024-09-16T16:45:04.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673007 2024-09-16T16:45:04.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949673007' 2024-09-16T16:45:04.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:45:04.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:45:04.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509478 2024-09-16T16:45:04.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509478 2024-09-16T16:45:04.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949673007 2-64424509478' 2024-09-16T16:45:04.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:45:04.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:45:04.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345949 2024-09-16T16:45:04.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345949 2024-09-16T16:45:04.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949673007 2-64424509478 3-85899345949' 2024-09-16T16:45:04.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:45:04.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:45:04.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182427 2024-09-16T16:45:04.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182427 2024-09-16T16:45:04.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949673007 2-64424509478 3-85899345949 4-107374182427' 2024-09-16T16:45:04.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:45:04.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T16:45:04.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018898 2024-09-16T16:45:04.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018898 2024-09-16T16:45:04.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949673007 2-64424509478 3-85899345949 4-107374182427 5-128849018898' 2024-09-16T16:45:04.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:45:04.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673007 2024-09-16T16:45:04.861 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:45:04.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:45:04.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673007 2024-09-16T16:45:04.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:45:04.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673007 2024-09-16T16:45:04.864 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673007 2024-09-16T16:45:04.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2024-09-16T16:45:04.865 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:45:05.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673007 -lt 42949673007 2024-09-16T16:45:05.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:45:05.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509478 2024-09-16T16:45:05.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:45:05.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:45:05.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509478 2024-09-16T16:45:05.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:45:05.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509478 2024-09-16T16:45:05.291 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509478 2024-09-16T16:45:05.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-09-16T16:45:05.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:45:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509478 -lt 64424509478 2024-09-16T16:45:05.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:45:05.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345949 2024-09-16T16:45:05.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:45:05.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:45:05.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345949 2024-09-16T16:45:05.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:45:05.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345949 2024-09-16T16:45:05.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345949' 2024-09-16T16:45:05.820 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345949 2024-09-16T16:45:05.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:45:06.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345947 -lt 85899345949 2024-09-16T16:45:06.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:45:07.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:45:07.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:45:07.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345949 -lt 85899345949 2024-09-16T16:45:07.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:45:07.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182427 2024-09-16T16:45:07.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:45:07.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:45:07.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182427 2024-09-16T16:45:07.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:45:07.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182427 2024-09-16T16:45:07.718 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182427 2024-09-16T16:45:07.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182427' 2024-09-16T16:45:07.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:45:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182427 -lt 107374182427 2024-09-16T16:45:08.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:45:08.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018898 2024-09-16T16:45:08.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:45:08.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T16:45:08.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018898 2024-09-16T16:45:08.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:45:08.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018898 2024-09-16T16:45:08.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018898' 2024-09-16T16:45:08.190 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018898 2024-09-16T16:45:08.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T16:45:08.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018899 -lt 128849018898 2024-09-16T16:45:08.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:45:08.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:45:08.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:45:09.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:45:09.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:45:09.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:45:09.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:45:09.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:45:09.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:45:09.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:45:09.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:45:09.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:45:09.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:45:09.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:45:09.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:45:10.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:45:10.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T16:45:10.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T16:45:10.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T16:45:10.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:45:10.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:45:10.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:45:10.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:45:10.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:45:10.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:45:10.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:45:10.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:45:10.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:45:10.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:45:11.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:45:11.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:45:11.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:45:11.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:45:11.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:45:11.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:45:11.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:45:11.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:45:11.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:45:11.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:45:11.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:45:11.717 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:45:11.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:45:12.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-09-16T16:45:12.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-09-16T16:45:12.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:45:12.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:45:12.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:45:12.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:45:12.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:45:12.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:45:12.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:45:12.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:45:12.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:45:12.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:45:12.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:45:13.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:45:13.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:45:13.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:45:13.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:45:13.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:45:13.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:45:13.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:45:13.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:45:13.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:45:13.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:45:13.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:45:13.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:45:13.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:45:14.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-16T16:45:14.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-16T16:45:14.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:45:14.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:45:14.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:45:14.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:45:14.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:45:14.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:45:14.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:45:14.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:45:14.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:45:14.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:45:14.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:45:14.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:45:14.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:45:14.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:45:14.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:45:15.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:45:15.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:45:15.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:45:15.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:45:15.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:45:15.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:45:15.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:45:15.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:45:15.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:45:15.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14 2024-09-16T16:45:15.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14 '!=' null 2024-09-16T16:45:15.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:45:15.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:45:15.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:45:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:45:16.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:45:16.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:45:16.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:45:16.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:45:16.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:45:16.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:45:16.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:45:16.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:45:16.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:45:16.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:45:16.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:45:17.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:45:17.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:45:17.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:45:17.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:45:17.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:45:17.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:45:17.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:45:17.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:45:17.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:45:17.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-09-16T16:45:17.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-09-16T16:45:17.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:45:17.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:45:17.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:45:17.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:45:17.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:45:17.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:45:17.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:45:17.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:45:17.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:45:17.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:45:17.745 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:45:18.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:45:18.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:45:18.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:45:18.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:45:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:45:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:45:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:45:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:45:18.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:45:18.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:45:18.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:45:18.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:45:18.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:45:19.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12 2024-09-16T16:45:19.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12 '!=' null 2024-09-16T16:45:19.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:45:19.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:45:19.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:45:19.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:45:19.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:45:19.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:45:19.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:45:19.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:45:19.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:45:19.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:45:19.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:45:19.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:45:19.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:45:19.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:45:19.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:45:20.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:45:20.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:45:20.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:45:20.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 replicated 200 0 0 0 2024-09-16T16:45:20.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-16T16:45:20.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-16T16:45:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-16T16:45:20.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2,4]/[1,2]' 2024-09-16T16:45:20.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-16T16:45:20.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:45:20.323 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:45:20.323 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2,4]/[1,2] ' 2024-09-16T16:45:20.323 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-16T16:45:20.323 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-16T16:45:20.324 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:45:20.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-09-16T16:45:20.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-09-16T16:45:20.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-16T16:45:20.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:45:20.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-16T16:45:20.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:45:20.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:45:20.331 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:45:20.331 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-16T16:45:20.331 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:45:20.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-16T16:45:20.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-16T16:45:20.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:45:20.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:45:20.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:45:20.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:45:20.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-16T16:45:20.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:45:20.341 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:45:20.341 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2,4]/[1,2] ' 2024-09-16T16:45:20.342 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-16T16:45:20.342 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-16T16:45:20.342 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:45:20.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-09-16T16:45:20.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-09-16T16:45:20.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-16T16:45:20.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:45:20.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-16T16:45:20.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:45:20.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:45:20.347 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:45:20.347 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-16T16:45:20.347 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:45:20.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-16T16:45:20.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-16T16:45:20.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:45:20.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:45:20.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:45:20.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:45:20.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-16T16:45:20.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-09-16T16:45:20.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T16:45:20.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T16:45:20.917 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T16:45:20.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-09-16T16:45:20.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:45:20.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:45:20.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:45:20.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:45:20.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:45:26.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:45:26.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-16T16:45:26.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:45:26.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:45:26.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:45:26.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:45:26.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:45:26.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:45:26.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:45:26.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:45:26.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:45:26.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:45:26.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:45:26.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:45:26.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:45:26.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:45:26.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:45:26.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:45:26.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:45:26.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:45:26.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:45:26.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:45:26.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:45:26.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:45:26.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:45:26.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:45:26.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:45:26.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:45:26.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:45:26.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:45:26.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-16T16:45:26.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-16T16:45:26.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-16T16:45:26.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-16T16:45:26.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:45:26.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:45:26.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:45:26.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:45:26.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:45:26.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:45:26.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:45:26.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:45:26.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:45:26.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:45:26.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:45:26.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:45:26.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:45:26.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:45:26.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:45:26.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:45:26.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:45:26.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:45:26.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:45:26.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:45:26.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:45:26.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:45:26.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:45:26.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:45:26.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:45:26.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:45:26.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:45:26.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:45:26.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-16T16:45:26.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:45:26.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:45:26.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:45:26.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274941 2024-09-16T16:45:26.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:45:26.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T16:45:26.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:45:26.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-16T16:45:26.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizedown td/osd-recovery-stats 2024-09-16T16:45:26.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:241: TEST_recovery_sizedown: local dir=td/osd-recovery-stats 2024-09-16T16:45:26.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:243: TEST_recovery_sizedown: run_mon td/osd-recovery-stats a 2024-09-16T16:45:26.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-16T16:45:26.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:45:26.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:45:26.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:45:26.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-16T16:45:26.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-16T16:45:26.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:45:26.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:45:26.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:45:26.692 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:45:26.692 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:45:26.692 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:45:26.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:45:26.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:45:26.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:45:26.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:45:26.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:45:26.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:45:26.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:45:26.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:45:26.894 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:45:26.894 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:45:26.894 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:45:26.894 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:45:26.894 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:45:26.894 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:45:26.894 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:45:26.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:45:26.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get fsid 2024-09-16T16:45:27.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:45:27.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:45:27.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:45:27.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:45:27.013 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:45:27.013 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:45:27.013 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:45:27.013 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:45:27.014 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:45:27.014 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:45:27.014 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:45:27.015 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:45:27.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:45:27.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get mon_host 2024-09-16T16:45:27.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:244: TEST_recovery_sizedown: run_mgr td/osd-recovery-stats x 2024-09-16T16:45:27.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-16T16:45:27.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:45:27.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:45:27.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:45:27.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-16T16:45:27.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:45:28.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:45:28.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:45:28.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:45:28.055 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:45:28.055 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:45:28.055 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:45:28.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:45:28.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:45:28.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:45:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:245: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 0 2024-09-16T16:45:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:45:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:45:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:45:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:45:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-16T16:45:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:45:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:45:28.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:45:28.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:45:28.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-16T16:45:28.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:45:28.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=667abae2-fafa-4a6a-885b-1ca862e89503 2024-09-16T16:45:28.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 667abae2-fafa-4a6a-885b-1ca862e89503' 2024-09-16T16:45:28.094 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 667abae2-fafa-4a6a-885b-1ca862e89503 2024-09-16T16:45:28.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:45:28.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoYOhmC6yTBhAAP8/KPZpodrEDpN/koCybmQ== 2024-09-16T16:45:28.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoYOhmC6yTBhAAP8/KPZpodrEDpN/koCybmQ=="}' 2024-09-16T16:45:28.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 667abae2-fafa-4a6a-885b-1ca862e89503 -i td/osd-recovery-stats/0/new.json 2024-09-16T16:45:28.493 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:45:28.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-16T16:45:28.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoYOhmC6yTBhAAP8/KPZpodrEDpN/koCybmQ== --osd-uuid 667abae2-fafa-4a6a-885b-1ca862e89503 2024-09-16T16:45:28.544 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:28.540+0000 7f2a867e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:45:28.546 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:28.544+0000 7f2a867e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:45:28.548 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:28.544+0000 7f2a867e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:45:28.548 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:28.544+0000 7f2a867e27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:45:28.548 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:28.544+0000 7f2a867e27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:45:28.548 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:28.544+0000 7f2a867e27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:45:28.624 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:28.620+0000 7f2a867e27c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-16T16:45:34.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-16T16:45:34.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:45:34.820 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:45:34.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:45:34.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:45:35.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:45:35.388 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:45:35.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:45:35.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:45:35.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:45:35.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:45:35.422 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:35.420+0000 7fac847cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:45:35.431 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:35.428+0000 7fac847cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:45:35.432 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:35.428+0000 7fac847cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:45:35.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:45:35.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:45:35.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:45:35.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:45:35.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:45:35.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:35.855 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:45:35.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:45:35.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:35.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:35.992 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:35.988+0000 7fac847cc7c0 -1 Falling back to public interface 2024-09-16T16:45:36.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:37.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:37.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:37.302 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:45:37.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:45:37.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:37.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:37.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:38.549 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:45:38.544+0000 7fac847cc7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:45:38.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:38.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:38.773 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:45:38.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:45:38.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:38.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:39.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:40.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:40.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:40.430 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:45:40.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:45:40.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:40.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:40.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:41.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:41.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:41.909 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:45:41.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:45:41.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:41.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:42.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:43.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:43.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:43.405 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:45:43.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:45:43.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:43.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:43.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:44.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:44.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:44.845 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:45:44.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:45:44.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:44.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:45.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:46.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:46.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:46.409 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:45:46.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:45:46.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:46.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:46.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:47.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:47.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:47.841 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:45:47.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:45:47.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:47.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:48.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:49.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:49.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:49.348 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:45:49.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:45:49.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:49.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:49.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:50.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:50.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:50.770 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:45:50.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:45:50.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:50.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:51.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:52.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:52.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:52.405 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:45:52.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:45:52.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:52.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:52.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:53.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:53.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:53.850 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:45:53.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:45:53.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:53.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:54.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:55.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:55.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:55.642 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:45:55.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:45:55.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:55.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:56.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:57.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:57.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:57.750 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:45:57.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:45:57.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:57.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:45:58.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:45:59.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:45:59.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:45:59.698 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:45:59.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:45:59.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:45:59.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:46:00.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:01.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:01.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:01.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:46:01.939 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:46:01.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:01.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:46:02.373 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2119657244,v1:127.0.0.1:6803/2119657244] [v2:127.0.0.1:6804/2119657244,v1:127.0.0.1:6805/2119657244] exists,up 667abae2-fafa-4a6a-885b-1ca862e89503 2024-09-16T16:46:02.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:46:02.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:46:02.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:46:02.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:246: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 1 2024-09-16T16:46:02.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:46:02.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:46:02.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:46:02.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:46:02.378 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:46:02.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:46:02.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-16T16:46:02.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:46:02.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=97befb7b-d16a-4437-adeb-820c467c0bdc 2024-09-16T16:46:02.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 97befb7b-d16a-4437-adeb-820c467c0bdc' 2024-09-16T16:46:02.381 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 97befb7b-d16a-4437-adeb-820c467c0bdc 2024-09-16T16:46:02.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:46:02.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKYOhm6l+dFxAA9H5se9//VeSfaUPrV2fELQ== 2024-09-16T16:46:02.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKYOhm6l+dFxAA9H5se9//VeSfaUPrV2fELQ=="}' 2024-09-16T16:46:02.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 97befb7b-d16a-4437-adeb-820c467c0bdc -i td/osd-recovery-stats/1/new.json 2024-09-16T16:46:02.939 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:46:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-16T16:46:02.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKYOhm6l+dFxAA9H5se9//VeSfaUPrV2fELQ== --osd-uuid 97befb7b-d16a-4437-adeb-820c467c0bdc 2024-09-16T16:46:02.993 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:02.988+0000 7fe94624b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:03.019 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:03.016+0000 7fe94624b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:03.021 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:03.016+0000 7fe94624b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:03.022 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:03.016+0000 7fe94624b7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:46:03.022 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:03.016+0000 7fe94624b7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:46:03.022 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:03.016+0000 7fe94624b7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:46:03.127 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:03.124+0000 7fe94624b7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-16T16:46:08.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-16T16:46:08.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:46:08.929 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:46:08.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:46:08.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:46:09.501 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:46:09.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:46:09.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:46:09.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:46:09.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:46:09.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:46:09.534 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:09.532+0000 7f99f9f897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:09.567 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:09.564+0000 7f99f9f897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:09.568 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:09.564+0000 7f99f9f897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:09.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:46:09.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:46:09.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:46:09.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:46:09.925 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:46:09.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:46:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:46:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:09.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:10.136 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:10.132+0000 7f99f9f897c0 -1 Falling back to public interface 2024-09-16T16:46:10.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:11.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:11.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:11.359 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:46:11.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:46:11.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:11.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:11.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:12.685 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:12.680+0000 7f99f9f897c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:46:12.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:12.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:12.776 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:46:12.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:46:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:12.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:13.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:14.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:14.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:14.398 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:46:14.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:46:14.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:14.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:14.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:15.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:15.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:15.895 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:46:15.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:46:15.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:15.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:16.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:17.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:17.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:17.518 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:46:17.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:46:17.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:17.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:18.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:19.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:19.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:19.066 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:46:19.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:46:19.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:19.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:19.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:20.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:20.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:20.607 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:46:20.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:46:20.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:20.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:21.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:22.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:22.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:22.040 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:46:22.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:46:22.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:22.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:22.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:23.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:23.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:23.653 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:46:23.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:46:23.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:23.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:24.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:25.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:25.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:25.095 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:46:25.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:46:25.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:25.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:25.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:26.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:26.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:26.521 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:46:26.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:46:26.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:26.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:27.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:28.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:28.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:28.066 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:46:28.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:46:28.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:28.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:29.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:30.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:30.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:30.096 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:46:30.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:46:30.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:30.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:30.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:31.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:31.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:31.880 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:46:31.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:46:31.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:31.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:32.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:33.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:33.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:33.685 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:46:33.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:46:33.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:33.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:34.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:35.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:35.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:35.736 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:46:35.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:46:35.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:35.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:36.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:37.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:37.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:37.631 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:46:37.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:46:37.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:37.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:46:38.101 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/71965354,v1:127.0.0.1:6811/71965354] [v2:127.0.0.1:6812/71965354,v1:127.0.0.1:6813/71965354] exists,up 97befb7b-d16a-4437-adeb-820c467c0bdc 2024-09-16T16:46:38.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:46:38.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:46:38.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:46:38.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:247: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 2 2024-09-16T16:46:38.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:46:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:46:38.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:46:38.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:46:38.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:46:38.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:46:38.106 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:46:38.106 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:46:38.106 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:46:38.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:46:38.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:46:38.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-16T16:46:38.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:46:38.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58f0619e-9fbb-4827-8f64-5c15c2a9cb0f 2024-09-16T16:46:38.109 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 58f0619e-9fbb-4827-8f64-5c15c2a9cb0f 2024-09-16T16:46:38.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 58f0619e-9fbb-4827-8f64-5c15c2a9cb0f' 2024-09-16T16:46:38.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:46:38.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuYOhmL0DDBxAANJ+95uDYgz9leVz4i8CoLg== 2024-09-16T16:46:38.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuYOhmL0DDBxAANJ+95uDYgz9leVz4i8CoLg=="}' 2024-09-16T16:46:38.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58f0619e-9fbb-4827-8f64-5c15c2a9cb0f -i td/osd-recovery-stats/2/new.json 2024-09-16T16:46:38.776 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:46:38.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-16T16:46:38.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuYOhmL0DDBxAANJ+95uDYgz9leVz4i8CoLg== --osd-uuid 58f0619e-9fbb-4827-8f64-5c15c2a9cb0f 2024-09-16T16:46:38.833 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:38.827+0000 7f9713b1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:38.835 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:38.831+0000 7f9713b1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:38.836 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:38.831+0000 7f9713b1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:38.837 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:38.831+0000 7f9713b1a7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:46:38.837 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:38.831+0000 7f9713b1a7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:46:38.837 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:38.831+0000 7f9713b1a7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:46:38.934 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:38.927+0000 7f9713b1a7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-16T16:46:45.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-16T16:46:45.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:46:45.063 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T16:46:45.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T16:46:45.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:46:45.627 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T16:46:45.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T16:46:45.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:46:45.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:46:45.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:46:45.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:46:45.661 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:45.655+0000 7f9041c5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:45.663 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:45.659+0000 7f9041c5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:45.665 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:45.659+0000 7f9041c5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:46:46.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T16:46:46.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:46:46.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T16:46:46.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:46:46.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:46:46.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:46.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:46:46.111 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:46:46.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:46.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:46:46.260 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:46.255+0000 7f9041c5f7c0 -1 Falling back to public interface 2024-09-16T16:46:46.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:46:47.560 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:46:47.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:47.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:46:48.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:48.891 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:46:48.887+0000 7f9041c5f7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T16:46:49.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:49.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:49.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:46:49.031 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:46:49.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:49.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:46:49.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:50.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:50.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:50.581 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:46:50.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:46:50.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:50.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:46:51.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:52.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:52.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:52.024 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:46:52.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:46:52.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:52.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:46:52.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:53.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:53.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:53.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:46:53.517 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:46:53.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:53.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:46:53.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:54.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:54.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:54.982 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:46:54.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:46:54.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:54.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:46:55.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:56.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:56.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:56.533 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:46:56.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:46:56.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:56.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:46:57.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:58.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:58.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:58.009 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:46:58.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:46:58.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:58.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:46:58.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:46:59.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:46:59.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:46:59.443 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:46:59.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:46:59.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:46:59.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:46:59.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:00.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:00.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:00.903 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:47:00.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:47:00.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:00.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:47:01.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:02.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:02.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:02.446 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:47:02.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:47:02.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:02.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:47:03.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:04.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:04.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:04.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:47:04.215 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:47:04.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:04.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:47:04.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:05.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:05.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:05.979 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:47:05.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:47:05.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:05.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:47:06.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:07.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:07.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:07.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:47:07.999 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:47:07.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:07.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:47:08.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:09.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:09.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:09.752 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:47:09.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:47:09.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:09.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:47:10.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:11.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:11.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:11.487 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:47:11.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:47:11.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:11.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:47:12.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:13.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:13.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:13.248 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:47:13.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:47:13.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:13.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:47:13.713 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1468914055,v1:127.0.0.1:6819/1468914055] [v2:127.0.0.1:6820/1468914055,v1:127.0.0.1:6821/1468914055] exists,up 58f0619e-9fbb-4827-8f64-5c15c2a9cb0f 2024-09-16T16:47:13.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:248: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 3 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:47:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:47:13.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-16T16:47:13.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-16T16:47:13.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:47:13.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:47:13.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:47:13.717 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:47:13.717 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:47:13.717 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:47:13.719 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:47:13.719 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:47:13.719 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:47:13.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:47:13.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:47:13.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:47:13.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:47:13.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:47:13.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:47:13.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:47:13.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:47:13.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:47:13.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:47:13.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:47:13.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:47:13.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:47:13.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-16T16:47:13.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:47:13.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2b5bdb9a-256b-4d0f-afeb-0c416947d291 2024-09-16T16:47:13.721 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 2b5bdb9a-256b-4d0f-afeb-0c416947d291 2024-09-16T16:47:13.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2b5bdb9a-256b-4d0f-afeb-0c416947d291' 2024-09-16T16:47:13.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:47:13.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARYehmnYEvLBAAQOGgpJWraADi5wphqdYWzw== 2024-09-16T16:47:13.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARYehmnYEvLBAAQOGgpJWraADi5wphqdYWzw=="}' 2024-09-16T16:47:13.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2b5bdb9a-256b-4d0f-afeb-0c416947d291 -i td/osd-recovery-stats/3/new.json 2024-09-16T16:47:14.315 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:47:14.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-16T16:47:14.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARYehmnYEvLBAAQOGgpJWraADi5wphqdYWzw== --osd-uuid 2b5bdb9a-256b-4d0f-afeb-0c416947d291 2024-09-16T16:47:14.368 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:14.366+0000 7fc93ca3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:14.405 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:14.402+0000 7fc93ca3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:14.406 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:14.402+0000 7fc93ca3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:14.407 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:14.406+0000 7fc93ca3b7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:47:14.407 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:14.406+0000 7fc93ca3b7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:47:14.407 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:14.406+0000 7fc93ca3b7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:47:14.479 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:14.478+0000 7fc93ca3b7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-16T16:47:20.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-16T16:47:20.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:47:20.886 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T16:47:20.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T16:47:20.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:47:21.452 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T16:47:21.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T16:47:21.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:47:21.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:47:21.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:47:21.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:47:21.485 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:21.482+0000 7f605f74f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:21.495 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:21.494+0000 7f605f74f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:21.497 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:21.494+0000 7f605f74f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:21.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T16:47:21.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:47:21.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T16:47:21.918 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:47:21.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:47:21.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:47:21.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:21.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:47:21.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:21.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:22.080 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:22.078+0000 7f605f74f7c0 -1 Falling back to public interface 2024-09-16T16:47:22.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:23.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:23.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:23.377 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:47:23.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:47:23.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:23.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:23.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:24.613 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:24.610+0000 7f605f74f7c0 -1 osd.3 0 log_to_monitors true 2024-09-16T16:47:24.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:24.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:24.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:47:24.794 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:47:24.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:24.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:25.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:26.239 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:47:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:47:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:26.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:26.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:27.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:27.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:27.740 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:47:27.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:47:27.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:27.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:28.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:29.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:29.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:29.177 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:47:29.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:47:29.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:29.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:29.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:30.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:30.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:30.723 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:47:30.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:47:30.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:30.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:31.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:32.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:32.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:32.210 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:47:32.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:47:32.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:32.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:32.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:33.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:33.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:33.759 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:47:33.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:47:33.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:33.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:34.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:35.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:35.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:35.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:47:35.239 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:47:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:35.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:36.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:36.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:36.682 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:47:36.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:47:36.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:36.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:37.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:38.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:38.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:38.196 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:47:38.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:47:38.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:38.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:39.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:40.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:40.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:40.132 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:47:40.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:47:40.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:40.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:40.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:41.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:41.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:41.947 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:47:41.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:47:41.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:41.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:42.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:43.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:43.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:43.648 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:47:43.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:47:43.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:43.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:44.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:45.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:45.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:45.403 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:47:45.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:47:45.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:45.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:46.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:47.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:47.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:47.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:47:47.163 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:47:47.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:47.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:47.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:48.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:48.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:48.877 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:47:48.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:47:48.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:48.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:47:49.349 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2942624876,v1:127.0.0.1:6827/2942624876] [v2:127.0.0.1:6828/2942624876,v1:127.0.0.1:6829/2942624876] exists,up 2b5bdb9a-256b-4d0f-afeb-0c416947d291 2024-09-16T16:47:49.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:47:49.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:47:49.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:47:49.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:249: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 4 2024-09-16T16:47:49.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:47:49.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:47:49.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T16:47:49.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:47:49.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-16T16:47:49.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:47:49.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:47:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-16T16:47:49.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:47:49.355 INFO:tasks.workunit.client.0.smithi039.stdout:add osd4 b7bad11a-9a40-4740-b545-55b54c1283e8 2024-09-16T16:47:49.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b7bad11a-9a40-4740-b545-55b54c1283e8 2024-09-16T16:47:49.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 b7bad11a-9a40-4740-b545-55b54c1283e8' 2024-09-16T16:47:49.355 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:47:49.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1Yehmmn5gFhAAkgEMnkL4lB7DX6yfNcNb0Q== 2024-09-16T16:47:49.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1Yehmmn5gFhAAkgEMnkL4lB7DX6yfNcNb0Q=="}' 2024-09-16T16:47:49.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b7bad11a-9a40-4740-b545-55b54c1283e8 -i td/osd-recovery-stats/4/new.json 2024-09-16T16:47:49.969 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:47:49.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-16T16:47:49.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1Yehmmn5gFhAAkgEMnkL4lB7DX6yfNcNb0Q== --osd-uuid b7bad11a-9a40-4740-b545-55b54c1283e8 2024-09-16T16:47:50.021 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:50.018+0000 7ff5d34137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:50.024 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:50.022+0000 7ff5d34137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:50.025 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:50.022+0000 7ff5d34137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:50.026 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:50.022+0000 7ff5d34137c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:47:50.026 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:50.022+0000 7ff5d34137c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:47:50.026 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:50.022+0000 7ff5d34137c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:47:50.128 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:50.126+0000 7ff5d34137c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-16T16:47:56.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-16T16:47:56.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:47:56.284 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd4 key to auth repository 2024-09-16T16:47:56.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T16:47:56.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:47:56.860 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.4 2024-09-16T16:47:56.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T16:47:56.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:47:56.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:47:56.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:47:56.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:47:56.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:56.893+0000 7f33d54d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:56.900 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:56.897+0000 7f33d54d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:56.901 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:56.897+0000 7f33d54d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:47:57.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T16:47:57.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:47:57.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T16:47:57.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:47:57.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:47:57.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:57.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:47:57.318 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:47:57.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:57.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:47:57.456 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:47:57.453+0000 7f33d54d87c0 -1 Falling back to public interface 2024-09-16T16:47:57.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:47:58.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:47:58.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:47:58.737 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:47:58.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:47:58.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:47:58.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:47:59.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:00.097+0000 7f33d54d87c0 -1 osd.4 0 log_to_monitors true 2024-09-16T16:48:00.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:00.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:00.190 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:48:00.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:48:00.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:00.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:00.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:01.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:01.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:01.851 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:48:01.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:48:01.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:01.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:02.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:03.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:03.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:03.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:48:03.278 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:48:03.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:03.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:03.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:04.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:04.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:04.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:48:04.924 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:48:04.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:04.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:05.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:06.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:06.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:06.474 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:48:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:48:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:07.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:08.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:08.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:08.007 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:48:08.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:48:08.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:08.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:08.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:09.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:09.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:09.467 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:48:09.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:48:09.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:09.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:10.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:11.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:11.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:11.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:48:11.014 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:48:11.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:11.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:11.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:12.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:12.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:12.513 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:48:12.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:48:12.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:12.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:13.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:14.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:14.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:14.072 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:48:14.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:48:14.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:14.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:14.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:15.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:15.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:15.850 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:48:15.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:48:15.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:15.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:16.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:17.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:17.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:17.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:48:17.695 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:48:17.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:17.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:18.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:19.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:19.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:19.978 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:48:19.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:48:19.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:19.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:21.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:22.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:22.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:22.089 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:48:22.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:48:22.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:22.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:22.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:23.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:23.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:23.932 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:48:23.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:48:23.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:23.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:24.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:25.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:25.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:25.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:48:25.495 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:48:25.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:25.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:48:25.956 INFO:tasks.workunit.client.0.smithi039.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2666263334,v1:127.0.0.1:6835/2666263334] [v2:127.0.0.1:6836/2666263334,v1:127.0.0.1:6837/2666263334] exists,up b7bad11a-9a40-4740-b545-55b54c1283e8 2024-09-16T16:48:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:48:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:48:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:48:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:250: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 5 2024-09-16T16:48:25.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:48:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:48:25.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:48:25.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:48:25.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:48:25.961 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:48:25.961 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:48:25.961 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:48:25.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:48:25.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:48:25.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:48:25.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:48:25.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:48:25.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:48:25.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:48:25.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:48:25.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:48:25.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:48:25.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:48:25.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:48:25.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:48:25.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-16T16:48:25.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:48:25.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c0ae7132-8c31-4098-ba16-673f5c97eb2e 2024-09-16T16:48:25.964 INFO:tasks.workunit.client.0.smithi039.stdout:add osd5 c0ae7132-8c31-4098-ba16-673f5c97eb2e 2024-09-16T16:48:25.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 c0ae7132-8c31-4098-ba16-673f5c97eb2e' 2024-09-16T16:48:25.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:48:25.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZYehmIJm3OhAAQhib4t+gf0Pp0sjXCq9RvA== 2024-09-16T16:48:25.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZYehmIJm3OhAAQhib4t+gf0Pp0sjXCq9RvA=="}' 2024-09-16T16:48:25.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c0ae7132-8c31-4098-ba16-673f5c97eb2e -i td/osd-recovery-stats/5/new.json 2024-09-16T16:48:26.603 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:48:26.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-16T16:48:26.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZYehmIJm3OhAAQhib4t+gf0Pp0sjXCq9RvA== --osd-uuid c0ae7132-8c31-4098-ba16-673f5c97eb2e 2024-09-16T16:48:26.655 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:26.653+0000 7fda571c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:48:26.657 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:26.653+0000 7fda571c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:48:26.659 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:26.657+0000 7fda571c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:48:26.659 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:26.657+0000 7fda571c87c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:48:26.659 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:26.657+0000 7fda571c87c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:48:26.659 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:26.657+0000 7fda571c87c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:48:26.766 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:26.761+0000 7fda571c87c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-16T16:48:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-16T16:48:32.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:48:32.901 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd5 key to auth repository 2024-09-16T16:48:32.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T16:48:32.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:48:33.458 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.5 2024-09-16T16:48:33.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T16:48:33.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:48:33.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:48:33.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:48:33.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:48:33.491 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:33.489+0000 7f717e8b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:48:33.493 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:33.489+0000 7f717e8b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:48:33.495 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:33.493+0000 7f717e8b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:48:33.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T16:48:33.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:48:33.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T16:48:33.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:48:33.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:48:33.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:33.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:48:33.910 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:48:33.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:33.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:34.079 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:34.077+0000 7f717e8b47c0 -1 Falling back to public interface 2024-09-16T16:48:34.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:35.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:35.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:35.337 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:48:35.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:48:35.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:35.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:35.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:36.694 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:48:36.689+0000 7f717e8b47c0 -1 osd.5 0 log_to_monitors true 2024-09-16T16:48:36.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:36.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:36.755 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:48:36.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:48:36.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:36.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:37.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:38.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:38.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:38.382 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:48:38.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:48:38.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:38.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:38.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:39.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:39.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:39.867 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:48:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:48:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:40.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:41.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:41.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:41.297 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:48:41.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:48:41.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:41.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:41.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:42.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:42.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:42.845 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:48:42.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:48:42.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:42.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:43.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:44.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:44.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:44.300 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:48:44.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:48:44.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:44.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:44.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:45.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:45.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:45.800 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:48:45.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:48:45.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:45.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:46.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:47.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:47.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:47.246 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:48:47.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:48:47.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:47.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:47.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:48.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:48.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:48.777 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:48:48.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:48:48.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:48.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:49.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:50.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:50.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:50.241 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:48:50.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:48:50.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:50.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:51.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:52.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:52.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:52.043 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:48:52.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:48:52.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:52.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:52.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:53.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:53.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:53.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:48:53.880 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:48:53.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:53.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:54.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:55.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:55.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:55.910 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:48:55.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:48:55.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:55.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:56.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:57.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:57.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:57.696 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:48:57.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:48:57.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:57.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:48:58.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:48:59.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:48:59.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:48:59.513 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:48:59.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:48:59.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:48:59.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:49:00.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:49:01.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:49:01.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:49:01.229 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:49:01.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:49:01.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:49:01.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:49:01.665 INFO:tasks.workunit.client.0.smithi039.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/749379797,v1:127.0.0.1:6843/749379797] [v2:127.0.0.1:6844/749379797,v1:127.0.0.1:6845/749379797] exists,up c0ae7132-8c31-4098-ba16-673f5c97eb2e 2024-09-16T16:49:01.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:49:01.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:49:01.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:49:01.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:252: TEST_recovery_sizedown: create_pool test 1 1 2024-09-16T16:49:01.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-16T16:49:02.339 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T16:49:02.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:49:03.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:253: TEST_recovery_sizedown: ceph osd pool set test size 4 2024-09-16T16:49:04.311 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 4 2024-09-16T16:49:04.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:255: TEST_recovery_sizedown: wait_for_clean 2024-09-16T16:49:04.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:49:04.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:49:04.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:49:04.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:49:04.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:49:04.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:49:04.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:49:04.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:49:04.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:49:04.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:49:04.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:49:04.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:49:04.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:49:04.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:49:04.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:49:04.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:49:04.931 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:49:04.931 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:49:04.931 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:49:04.931 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T16:49:04.931 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T16:49:04.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:49:04.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:04.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:49:05.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836523 2024-09-16T16:49:05.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836523 2024-09-16T16:49:05.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523' 2024-09-16T16:49:05.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:05.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:49:05.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672997 2024-09-16T16:49:05.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672997 2024-09-16T16:49:05.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949672997' 2024-09-16T16:49:05.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:05.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:49:05.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509466 2024-09-16T16:49:05.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509466 2024-09-16T16:49:05.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949672997 2-64424509466' 2024-09-16T16:49:05.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:05.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:49:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345941 2024-09-16T16:49:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345941 2024-09-16T16:49:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949672997 2-64424509466 3-85899345941' 2024-09-16T16:49:05.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:05.712 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:49:05.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182413 2024-09-16T16:49:05.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182413 2024-09-16T16:49:05.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949672997 2-64424509466 3-85899345941 4-107374182413' 2024-09-16T16:49:05.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:05.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T16:49:06.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018887 2024-09-16T16:49:06.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018887 2024-09-16T16:49:06.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949672997 2-64424509466 3-85899345941 4-107374182413 5-128849018887' 2024-09-16T16:49:06.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:06.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836523 2024-09-16T16:49:06.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:06.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:49:06.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836523 2024-09-16T16:49:06.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:06.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836523 2024-09-16T16:49:06.029 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836523 2024-09-16T16:49:06.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-09-16T16:49:06.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:49:06.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836521 -lt 21474836523 2024-09-16T16:49:06.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:49:07.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:49:07.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:49:07.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836523 -lt 21474836523 2024-09-16T16:49:07.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:07.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672997 2024-09-16T16:49:07.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:07.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:49:07.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672997 2024-09-16T16:49:07.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:07.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672997 2024-09-16T16:49:07.945 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672997 2024-09-16T16:49:07.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2024-09-16T16:49:07.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:49:08.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672997 -lt 42949672997 2024-09-16T16:49:08.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:08.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509466 2024-09-16T16:49:08.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:08.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:49:08.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509466 2024-09-16T16:49:08.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:08.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509466 2024-09-16T16:49:08.380 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509466 2024-09-16T16:49:08.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-09-16T16:49:08.381 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:49:08.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509466 -lt 64424509466 2024-09-16T16:49:08.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:08.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345941 2024-09-16T16:49:08.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:08.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:49:08.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345941 2024-09-16T16:49:08.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:08.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345941 2024-09-16T16:49:08.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-09-16T16:49:08.818 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345941 2024-09-16T16:49:08.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:49:09.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345941 -lt 85899345941 2024-09-16T16:49:09.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:09.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182413 2024-09-16T16:49:09.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:09.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:49:09.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182413 2024-09-16T16:49:09.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:09.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182413 2024-09-16T16:49:09.275 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182413 2024-09-16T16:49:09.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2024-09-16T16:49:09.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:49:09.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182413 -lt 107374182413 2024-09-16T16:49:09.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:09.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018887 2024-09-16T16:49:09.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:09.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T16:49:09.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018887 2024-09-16T16:49:09.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:09.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018887 2024-09-16T16:49:09.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-09-16T16:49:09.706 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018887 2024-09-16T16:49:09.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T16:49:10.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018888 -lt 128849018887 2024-09-16T16:49:10.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:49:10.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:49:10.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:49:10.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:49:10.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:49:10.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:49:10.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:49:10.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:49:10.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:49:10.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:49:10.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:49:11.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:49:11.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:49:11.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:49:11.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:49:11.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:49:11.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:49:11.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:49:11.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: seq 1 200 2024-09-16T16:49:11.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:11.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj1 /dev/null 2024-09-16T16:49:11.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:11.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj2 /dev/null 2024-09-16T16:49:12.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:12.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj3 /dev/null 2024-09-16T16:49:12.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:12.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj4 /dev/null 2024-09-16T16:49:12.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:12.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj5 /dev/null 2024-09-16T16:49:12.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:12.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj6 /dev/null 2024-09-16T16:49:12.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:12.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj7 /dev/null 2024-09-16T16:49:12.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:12.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj8 /dev/null 2024-09-16T16:49:12.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:12.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj9 /dev/null 2024-09-16T16:49:13.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:13.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj10 /dev/null 2024-09-16T16:49:13.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:13.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj11 /dev/null 2024-09-16T16:49:13.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:13.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj12 /dev/null 2024-09-16T16:49:13.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:13.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj13 /dev/null 2024-09-16T16:49:13.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:13.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj14 /dev/null 2024-09-16T16:49:13.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:13.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj15 /dev/null 2024-09-16T16:49:14.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:14.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj16 /dev/null 2024-09-16T16:49:14.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:14.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj17 /dev/null 2024-09-16T16:49:14.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:14.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj18 /dev/null 2024-09-16T16:49:14.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:14.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj19 /dev/null 2024-09-16T16:49:14.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:14.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj20 /dev/null 2024-09-16T16:49:14.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:14.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj21 /dev/null 2024-09-16T16:49:14.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:14.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj22 /dev/null 2024-09-16T16:49:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:15.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj23 /dev/null 2024-09-16T16:49:15.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:15.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj24 /dev/null 2024-09-16T16:49:15.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:15.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj25 /dev/null 2024-09-16T16:49:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:15.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj26 /dev/null 2024-09-16T16:49:15.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:15.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj27 /dev/null 2024-09-16T16:49:15.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:15.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj28 /dev/null 2024-09-16T16:49:16.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:16.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj29 /dev/null 2024-09-16T16:49:16.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:16.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj30 /dev/null 2024-09-16T16:49:16.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:16.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj31 /dev/null 2024-09-16T16:49:16.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:16.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj32 /dev/null 2024-09-16T16:49:16.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:16.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj33 /dev/null 2024-09-16T16:49:16.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:16.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj34 /dev/null 2024-09-16T16:49:17.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:17.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj35 /dev/null 2024-09-16T16:49:17.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:17.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj36 /dev/null 2024-09-16T16:49:17.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:17.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj37 /dev/null 2024-09-16T16:49:17.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:17.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj38 /dev/null 2024-09-16T16:49:17.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:17.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj39 /dev/null 2024-09-16T16:49:17.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:17.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj40 /dev/null 2024-09-16T16:49:17.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:17.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj41 /dev/null 2024-09-16T16:49:18.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:18.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj42 /dev/null 2024-09-16T16:49:18.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:18.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj43 /dev/null 2024-09-16T16:49:18.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:18.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj44 /dev/null 2024-09-16T16:49:18.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:18.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj45 /dev/null 2024-09-16T16:49:18.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:18.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj46 /dev/null 2024-09-16T16:49:18.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:18.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj47 /dev/null 2024-09-16T16:49:19.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:19.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj48 /dev/null 2024-09-16T16:49:19.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:19.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj49 /dev/null 2024-09-16T16:49:19.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:19.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj50 /dev/null 2024-09-16T16:49:19.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:19.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj51 /dev/null 2024-09-16T16:49:19.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:19.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj52 /dev/null 2024-09-16T16:49:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj53 /dev/null 2024-09-16T16:49:20.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:20.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj54 /dev/null 2024-09-16T16:49:20.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:20.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj55 /dev/null 2024-09-16T16:49:20.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:20.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj56 /dev/null 2024-09-16T16:49:20.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:20.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj57 /dev/null 2024-09-16T16:49:20.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:20.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj58 /dev/null 2024-09-16T16:49:21.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:21.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj59 /dev/null 2024-09-16T16:49:21.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:21.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj60 /dev/null 2024-09-16T16:49:21.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:21.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj61 /dev/null 2024-09-16T16:49:21.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:21.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj62 /dev/null 2024-09-16T16:49:21.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:21.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj63 /dev/null 2024-09-16T16:49:21.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:21.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj64 /dev/null 2024-09-16T16:49:21.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:21.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj65 /dev/null 2024-09-16T16:49:22.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:22.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj66 /dev/null 2024-09-16T16:49:22.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:22.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj67 /dev/null 2024-09-16T16:49:22.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:22.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj68 /dev/null 2024-09-16T16:49:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:22.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj69 /dev/null 2024-09-16T16:49:22.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:22.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj70 /dev/null 2024-09-16T16:49:22.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:22.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj71 /dev/null 2024-09-16T16:49:23.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:23.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj72 /dev/null 2024-09-16T16:49:23.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:23.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj73 /dev/null 2024-09-16T16:49:23.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:23.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj74 /dev/null 2024-09-16T16:49:23.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:23.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj75 /dev/null 2024-09-16T16:49:23.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:23.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj76 /dev/null 2024-09-16T16:49:23.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:23.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj77 /dev/null 2024-09-16T16:49:23.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:23.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj78 /dev/null 2024-09-16T16:49:24.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:24.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj79 /dev/null 2024-09-16T16:49:24.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:24.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj80 /dev/null 2024-09-16T16:49:24.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:24.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj81 /dev/null 2024-09-16T16:49:24.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:24.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj82 /dev/null 2024-09-16T16:49:24.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:24.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj83 /dev/null 2024-09-16T16:49:24.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:24.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj84 /dev/null 2024-09-16T16:49:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:25.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj85 /dev/null 2024-09-16T16:49:25.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:25.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj86 /dev/null 2024-09-16T16:49:25.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:25.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj87 /dev/null 2024-09-16T16:49:25.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:25.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj88 /dev/null 2024-09-16T16:49:25.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:25.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj89 /dev/null 2024-09-16T16:49:25.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:25.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj90 /dev/null 2024-09-16T16:49:25.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:25.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj91 /dev/null 2024-09-16T16:49:26.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:26.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj92 /dev/null 2024-09-16T16:49:26.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:26.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj93 /dev/null 2024-09-16T16:49:26.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:26.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj94 /dev/null 2024-09-16T16:49:26.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:26.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj95 /dev/null 2024-09-16T16:49:26.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:26.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj96 /dev/null 2024-09-16T16:49:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:26.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj97 /dev/null 2024-09-16T16:49:27.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:27.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj98 /dev/null 2024-09-16T16:49:27.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:27.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj99 /dev/null 2024-09-16T16:49:27.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:27.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj100 /dev/null 2024-09-16T16:49:27.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:27.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj101 /dev/null 2024-09-16T16:49:27.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:27.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj102 /dev/null 2024-09-16T16:49:27.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:27.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj103 /dev/null 2024-09-16T16:49:27.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:27.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj104 /dev/null 2024-09-16T16:49:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:28.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj105 /dev/null 2024-09-16T16:49:28.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:28.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj106 /dev/null 2024-09-16T16:49:28.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:28.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj107 /dev/null 2024-09-16T16:49:28.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:28.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj108 /dev/null 2024-09-16T16:49:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj109 /dev/null 2024-09-16T16:49:28.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:28.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj110 /dev/null 2024-09-16T16:49:28.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:28.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj111 /dev/null 2024-09-16T16:49:29.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:29.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj112 /dev/null 2024-09-16T16:49:29.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:29.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj113 /dev/null 2024-09-16T16:49:29.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:29.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj114 /dev/null 2024-09-16T16:49:29.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:29.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj115 /dev/null 2024-09-16T16:49:29.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:29.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj116 /dev/null 2024-09-16T16:49:29.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:29.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj117 /dev/null 2024-09-16T16:49:30.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:30.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj118 /dev/null 2024-09-16T16:49:30.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:30.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj119 /dev/null 2024-09-16T16:49:30.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:30.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj120 /dev/null 2024-09-16T16:49:30.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:30.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj121 /dev/null 2024-09-16T16:49:30.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:30.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj122 /dev/null 2024-09-16T16:49:30.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:30.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj123 /dev/null 2024-09-16T16:49:30.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:30.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj124 /dev/null 2024-09-16T16:49:31.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:31.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj125 /dev/null 2024-09-16T16:49:31.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:31.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj126 /dev/null 2024-09-16T16:49:31.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:31.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj127 /dev/null 2024-09-16T16:49:31.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:31.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj128 /dev/null 2024-09-16T16:49:31.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:31.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj129 /dev/null 2024-09-16T16:49:32.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:32.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj130 /dev/null 2024-09-16T16:49:32.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:32.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj131 /dev/null 2024-09-16T16:49:32.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:32.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj132 /dev/null 2024-09-16T16:49:32.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:32.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj133 /dev/null 2024-09-16T16:49:32.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:32.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj134 /dev/null 2024-09-16T16:49:32.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:32.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj135 /dev/null 2024-09-16T16:49:32.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:32.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj136 /dev/null 2024-09-16T16:49:33.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:33.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj137 /dev/null 2024-09-16T16:49:33.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:33.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj138 /dev/null 2024-09-16T16:49:33.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:33.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj139 /dev/null 2024-09-16T16:49:33.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:33.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj140 /dev/null 2024-09-16T16:49:33.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:33.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj141 /dev/null 2024-09-16T16:49:33.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:33.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj142 /dev/null 2024-09-16T16:49:34.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:34.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj143 /dev/null 2024-09-16T16:49:34.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:34.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj144 /dev/null 2024-09-16T16:49:34.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:34.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj145 /dev/null 2024-09-16T16:49:34.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:34.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj146 /dev/null 2024-09-16T16:49:34.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:34.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj147 /dev/null 2024-09-16T16:49:34.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:34.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj148 /dev/null 2024-09-16T16:49:34.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:34.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj149 /dev/null 2024-09-16T16:49:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj150 /dev/null 2024-09-16T16:49:35.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:35.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj151 /dev/null 2024-09-16T16:49:35.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:35.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj152 /dev/null 2024-09-16T16:49:35.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:35.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj153 /dev/null 2024-09-16T16:49:35.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:35.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj154 /dev/null 2024-09-16T16:49:35.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:35.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj155 /dev/null 2024-09-16T16:49:36.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:36.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj156 /dev/null 2024-09-16T16:49:36.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:36.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj157 /dev/null 2024-09-16T16:49:36.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:36.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj158 /dev/null 2024-09-16T16:49:36.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:36.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj159 /dev/null 2024-09-16T16:49:36.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:36.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj160 /dev/null 2024-09-16T16:49:36.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:36.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj161 /dev/null 2024-09-16T16:49:36.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:36.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj162 /dev/null 2024-09-16T16:49:37.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:37.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj163 /dev/null 2024-09-16T16:49:37.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:37.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj164 /dev/null 2024-09-16T16:49:37.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:37.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj165 /dev/null 2024-09-16T16:49:37.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:37.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj166 /dev/null 2024-09-16T16:49:37.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:37.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj167 /dev/null 2024-09-16T16:49:37.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:37.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj168 /dev/null 2024-09-16T16:49:38.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:38.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj169 /dev/null 2024-09-16T16:49:38.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:38.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj170 /dev/null 2024-09-16T16:49:38.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:38.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj171 /dev/null 2024-09-16T16:49:38.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:38.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj172 /dev/null 2024-09-16T16:49:38.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:38.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj173 /dev/null 2024-09-16T16:49:38.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:38.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj174 /dev/null 2024-09-16T16:49:38.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:38.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj175 /dev/null 2024-09-16T16:49:39.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:39.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj176 /dev/null 2024-09-16T16:49:39.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:39.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj177 /dev/null 2024-09-16T16:49:39.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:39.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj178 /dev/null 2024-09-16T16:49:39.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:39.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj179 /dev/null 2024-09-16T16:49:39.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:39.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj180 /dev/null 2024-09-16T16:49:40.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:40.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj181 /dev/null 2024-09-16T16:49:40.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:40.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj182 /dev/null 2024-09-16T16:49:40.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:40.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj183 /dev/null 2024-09-16T16:49:40.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:40.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj184 /dev/null 2024-09-16T16:49:40.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:40.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj185 /dev/null 2024-09-16T16:49:40.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:40.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj186 /dev/null 2024-09-16T16:49:40.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:40.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj187 /dev/null 2024-09-16T16:49:41.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:41.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj188 /dev/null 2024-09-16T16:49:41.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:41.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj189 /dev/null 2024-09-16T16:49:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj190 /dev/null 2024-09-16T16:49:41.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:41.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj191 /dev/null 2024-09-16T16:49:41.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:41.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj192 /dev/null 2024-09-16T16:49:41.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:41.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj193 /dev/null 2024-09-16T16:49:42.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:42.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj194 /dev/null 2024-09-16T16:49:42.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:42.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj195 /dev/null 2024-09-16T16:49:42.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:42.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj196 /dev/null 2024-09-16T16:49:42.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:42.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj197 /dev/null 2024-09-16T16:49:42.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:42.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj198 /dev/null 2024-09-16T16:49:42.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:42.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj199 /dev/null 2024-09-16T16:49:42.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-16T16:49:42.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj200 /dev/null 2024-09-16T16:49:43.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: get_primary test obj1 2024-09-16T16:49:43.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:49:43.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:49:43.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:49:43.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:49:43.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: local primary=1 2024-09-16T16:49:43.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: get_pg test obj1 2024-09-16T16:49:43.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-16T16:49:43.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-16T16:49:43.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T16:49:43.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-16T16:49:44.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: local PG=1.0 2024-09-16T16:49:44.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: get_osds test obj1 2024-09-16T16:49:44.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-09-16T16:49:44.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj1 2024-09-16T16:49:44.021 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test obj1 2024-09-16T16:49:44.021 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T16:49:44.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-16T16:49:44.478 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-09-16T16:49:44.478 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:49:44.478 INFO:tasks.workunit.client.0.smithi039.stderr:4' 2024-09-16T16:49:44.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 4 2024-09-16T16:49:44.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: local 'allosds=1 0 2 4' 2024-09-16T16:49:44.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:267: TEST_recovery_sizedown: ceph osd set norecover 2024-09-16T16:49:45.373 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is set 2024-09-16T16:49:45.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-16T16:49:45.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.1 2024-09-16T16:49:46.009 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already out. 2024-09-16T16:49:46.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-16T16:49:46.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.0 2024-09-16T16:49:46.822 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T16:49:46.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-16T16:49:46.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.2 2024-09-16T16:49:47.551 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already out. 2024-09-16T16:49:47.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-16T16:49:47.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.4 2024-09-16T16:49:48.835 INFO:tasks.workunit.client.0.smithi039.stderr:osd.4 is already out. 2024-09-16T16:49:48.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:273: TEST_recovery_sizedown: ceph osd pool set test size 2 2024-09-16T16:49:50.726 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-09-16T16:49:50.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:274: TEST_recovery_sizedown: ceph osd unset norecover 2024-09-16T16:49:52.251 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is unset 2024-09-16T16:49:52.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: get_primary test obj1 2024-09-16T16:49:52.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:49:52.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:49:52.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:49:52.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:49:52.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: ceph tell osd.3 debug kick_recovery_wq 0 2024-09-16T16:49:53.159 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T16:49:53.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:276: TEST_recovery_sizedown: sleep 2 2024-09-16T16:49:55.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:278: TEST_recovery_sizedown: wait_for_clean 2024-09-16T16:49:55.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:49:55.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:49:55.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:49:55.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:49:55.179 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:49:55.179 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:49:55.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:49:55.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:49:55.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:49:55.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:49:55.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:49:55.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:49:55.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:49:55.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:49:55.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:49:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:49:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:49:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:49:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:49:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T16:49:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T16:49:55.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:49:55.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:55.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:49:55.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836535 2024-09-16T16:49:55.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836535 2024-09-16T16:49:55.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836535' 2024-09-16T16:49:55.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:55.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:49:56.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673009 2024-09-16T16:49:56.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673009 2024-09-16T16:49:56.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836535 1-42949673009' 2024-09-16T16:49:56.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:56.233 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:49:56.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509478 2024-09-16T16:49:56.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509478 2024-09-16T16:49:56.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836535 1-42949673009 2-64424509478' 2024-09-16T16:49:56.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:56.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:49:56.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345953 2024-09-16T16:49:56.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345953 2024-09-16T16:49:56.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836535 1-42949673009 2-64424509478 3-85899345953' 2024-09-16T16:49:56.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:56.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:49:56.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182425 2024-09-16T16:49:56.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182425 2024-09-16T16:49:56.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836535 1-42949673009 2-64424509478 3-85899345953 4-107374182425' 2024-09-16T16:49:56.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:49:56.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T16:49:56.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018900 2024-09-16T16:49:56.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018900 2024-09-16T16:49:56.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836535 1-42949673009 2-64424509478 3-85899345953 4-107374182425 5-128849018900' 2024-09-16T16:49:56.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:56.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836535 2024-09-16T16:49:56.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:56.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:49:56.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:56.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836535 2024-09-16T16:49:56.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836535 2024-09-16T16:49:56.873 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836535 2024-09-16T16:49:56.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-09-16T16:49:56.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:49:57.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836535 -lt 21474836535 2024-09-16T16:49:57.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:57.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673009 2024-09-16T16:49:57.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:57.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:49:57.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673009 2024-09-16T16:49:57.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:57.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673009 2024-09-16T16:49:57.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2024-09-16T16:49:57.301 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673009 2024-09-16T16:49:57.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:49:57.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673009 -lt 42949673009 2024-09-16T16:49:57.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:57.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509478 2024-09-16T16:49:57.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:57.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:49:57.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509478 2024-09-16T16:49:57.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:57.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509478 2024-09-16T16:49:57.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-09-16T16:49:57.781 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509478 2024-09-16T16:49:57.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:49:58.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509478 -lt 64424509478 2024-09-16T16:49:58.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:58.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345953 2024-09-16T16:49:58.199 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:58.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:49:58.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345953 2024-09-16T16:49:58.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:58.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345953 2024-09-16T16:49:58.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345953' 2024-09-16T16:49:58.204 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345953 2024-09-16T16:49:58.204 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:49:58.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345953 -lt 85899345953 2024-09-16T16:49:58.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:58.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182425 2024-09-16T16:49:58.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:58.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:49:58.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182425 2024-09-16T16:49:58.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:58.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182425 2024-09-16T16:49:58.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182425' 2024-09-16T16:49:58.698 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182425 2024-09-16T16:49:58.698 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:49:59.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182425 -lt 107374182425 2024-09-16T16:49:59.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:49:59.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018900 2024-09-16T16:49:59.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:49:59.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T16:49:59.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018900 2024-09-16T16:49:59.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:49:59.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018900 2024-09-16T16:49:59.317 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018900 2024-09-16T16:49:59.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018900' 2024-09-16T16:49:59.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T16:49:59.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018900 -lt 128849018900 2024-09-16T16:49:59.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:49:59.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:49:59.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:50:00.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:50:00.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:50:00.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:50:00.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:50:00.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:50:00.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:50:00.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:50:00.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:50:00.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:50:00.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:50:00.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:50:00.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:50:01.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:50:01.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T16:50:01.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T16:50:01.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T16:50:01.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:50:01.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:50:01.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:50:01.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:50:01.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:50:01.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:50:01.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:50:01.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:50:01.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:50:01.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:50:02.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:50:02.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:50:02.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:50:02.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:50:02.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:50:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:50:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:50:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:50:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:50:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:50:02.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:50:02.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:50:02.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:50:03.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-16T16:50:03.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-16T16:50:03.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:50:03.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:50:03.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:50:03.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:50:03.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:50:03.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:50:03.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:50:03.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:50:03.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:50:03.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:50:03.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:50:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:50:03.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:50:03.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:50:03.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:50:04.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:50:04.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:50:04.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:50:04.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:50:04.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:50:04.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:50:04.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:50:04.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:50:04.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:50:05.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-16T16:50:05.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-16T16:50:05.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:50:05.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:50:05.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:50:05.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:50:05.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:50:05.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:50:05.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:50:05.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:50:05.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:50:05.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:50:05.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:50:05.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:50:05.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:50:05.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:50:05.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:50:06.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:50:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:50:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:50:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:50:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:50:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:50:06.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:50:06.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:50:06.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:50:06.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-16T16:50:06.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-16T16:50:06.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:50:06.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:50:06.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:50:07.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:50:07.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:50:07.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:50:07.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:50:07.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:50:07.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:50:07.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:50:07.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:50:07.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:50:07.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:50:07.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:50:07.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:50:08.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:50:08.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:50:08.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:50:08.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:50:08.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:50:08.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:50:08.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:50:08.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:50:08.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:50:08.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-09-16T16:50:08.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-09-16T16:50:08.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:50:08.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:50:08.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:50:08.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:50:08.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:50:08.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:50:08.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:50:08.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:50:08.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:50:08.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:50:08.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:50:09.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:50:09.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:50:09.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:50:09.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:50:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:50:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:50:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:50:09.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:50:09.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:50:09.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:50:09.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:50:09.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:50:09.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:50:10.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-16T16:50:10.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-16T16:50:10.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:50:10.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:50:10.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:50:10.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:50:10.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:50:10.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:50:10.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:50:10.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:50:10.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:50:10.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:50:10.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:50:11.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:50:11.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:50:11.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:50:11.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:50:11.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:50:11.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:50:11.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:50:11.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:50:11.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:50:11.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:50:11.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:50:11.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:50:11.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:50:12.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-09-16T16:50:12.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-09-16T16:50:12.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:50:12.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:50:12.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:50:12.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:50:12.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:50:12.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:50:12.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:50:12.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:50:12.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:50:12.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:50:12.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:50:13.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:50:13.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:50:13.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:50:13.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:50:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:50:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:50:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:50:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:50:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:50:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:50:13.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:50:13.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:50:13.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:50:14.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-16T16:50:14.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-16T16:50:14.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:50:14.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:50:14.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:50:14.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:50:14.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:50:14.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:50:14.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:50:14.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:50:14.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:50:14.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:50:14.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:50:14.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:50:14.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:50:14.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:50:14.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:50:15.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:50:15.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:50:15.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:50:15.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: get_primary test obj1 2024-09-16T16:50:15.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:50:15.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:50:15.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:50:15.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:50:15.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: primary=3 2024-09-16T16:50:15.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: expr 200 '*' 2 2024-09-16T16:50:15.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: local misplaced=400 2024-09-16T16:50:15.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:284: TEST_recovery_sizedown: local log=td/osd-recovery-stats/osd.3.log 2024-09-16T16:50:15.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:285: TEST_recovery_sizedown: check td/osd-recovery-stats 1.0 3 replicated 0 0 400 0 2024-09-16T16:50:15.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-16T16:50:15.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-16T16:50:15.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=3 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=0 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.3.log 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-09-16T16:50:15.956 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-16T16:50:15.957 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-16T16:50:15.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[3,5]/[3]' 2024-09-16T16:50:15.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-16T16:50:15.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:50:15.979 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-09-16T16:50:15.979 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [3,5]/[3] ' 2024-09-16T16:50:15.979 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-16T16:50:15.980 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-16T16:50:15.980 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:50:15.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=0 2024-09-16T16:50:15.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 0 0 2024-09-16T16:50:15.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-16T16:50:15.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:50:15.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-16T16:50:15.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:50:15.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:50:15.985 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-09-16T16:50:15.985 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-16T16:50:15.985 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:50:15.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-16T16:50:15.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-16T16:50:15.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:50:15.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:50:15.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:50:15.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:50:15.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-16T16:50:15.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:50:15.996 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-09-16T16:50:15.996 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [3,5]/[3] ' 2024-09-16T16:50:15.996 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-16T16:50:15.996 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-16T16:50:15.997 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:50:16.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-09-16T16:50:16.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-09-16T16:50:16.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-09-16T16:50:16.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:50:16.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-09-16T16:50:16.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:50:16.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:50:16.001 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-09-16T16:50:16.001 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-16T16:50:16.001 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:50:16.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-16T16:50:16.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-16T16:50:16.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:50:16.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:50:16.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:50:16.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:50:16.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-16T16:50:16.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-09-16T16:50:16.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: tail -1 2024-09-16T16:50:16.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-16T16:50:16.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: UPACT='[3,5]/[3]' 2024-09-16T16:50:16.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-09-16T16:50:16.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep -F ' [3,5]/[3] ' 2024-09-16T16:50:16.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: head -1 2024-09-16T16:50:16.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:50:16.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: FIRST=200 2024-09-16T16:50:16.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:291: TEST_recovery_sizedown: below_margin 200 200 2024-09-16T16:50:16.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-16T16:50:16.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:50:16.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-16T16:50:16.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:50:16.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-09-16T16:50:16.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: tail -1 2024-09-16T16:50:16.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:50:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: LAST=0 2024-09-16T16:50:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:293: TEST_recovery_sizedown: above_margin 0 0 2024-09-16T16:50:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:50:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:50:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:50:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:50:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:295: TEST_recovery_sizedown: delete_pool test 2024-09-16T16:50:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T16:50:16.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T16:50:16.574 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T16:50:16.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:296: TEST_recovery_sizedown: kill_daemons td/osd-recovery-stats 2024-09-16T16:50:16.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:50:16.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:50:16.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:50:16.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:50:16.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:50:21.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:50:21.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-16T16:50:21.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:50:21.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:50:21.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:50:21.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:50:21.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:50:21.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:50:21.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:50:21.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:50:21.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:50:21.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:50:21.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:50:21.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:50:21.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:50:21.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:50:21.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:50:21.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:50:21.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:50:21.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:50:21.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:50:21.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:50:21.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:50:21.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:50:21.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:50:21.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:50:21.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:50:21.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:50:21.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:50:21.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:50:21.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-16T16:50:21.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-16T16:50:21.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-16T16:50:21.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-16T16:50:21.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:50:21.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:50:21.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:50:21.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:50:21.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:50:21.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:50:21.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:50:21.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:50:21.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:50:21.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:50:21.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:50:21.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:50:21.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:50:21.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:50:21.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:50:21.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:50:21.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:50:21.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:50:21.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:50:21.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:50:21.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:50:21.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:50:21.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:50:21.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:50:21.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:50:21.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:50:21.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:50:21.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:50:21.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-16T16:50:21.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:50:21.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:50:21.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274941 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizeup td/osd-recovery-stats 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:186: TEST_recovery_sizeup: local dir=td/osd-recovery-stats 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:188: TEST_recovery_sizeup: run_mon td/osd-recovery-stats a 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-16T16:50:21.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-16T16:50:22.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:50:22.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:50:22.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:50:22.365 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:50:22.365 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:50:22.366 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:50:22.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:50:22.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:50:22.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:50:22.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:50:22.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:50:22.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:50:22.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:50:22.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:50:22.595 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:50:22.595 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:50:22.595 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:50:22.596 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:50:22.596 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:50:22.596 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:50:22.596 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:50:22.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:50:22.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get fsid 2024-09-16T16:50:22.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:50:22.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:50:22.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:50:22.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:50:22.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:50:22.715 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:50:22.715 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:50:22.715 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:50:22.716 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:50:22.716 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:50:22.716 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:50:22.716 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:50:22.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:50:22.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get mon_host 2024-09-16T16:50:22.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:189: TEST_recovery_sizeup: run_mgr td/osd-recovery-stats x 2024-09-16T16:50:22.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-16T16:50:22.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:50:22.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:50:22.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:50:22.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-16T16:50:22.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:50:23.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:50:23.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:50:23.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:50:23.743 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:50:23.743 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:50:23.744 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:50:23.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:50:23.744 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:50:23.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:50:23.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:190: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 0 2024-09-16T16:50:23.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:50:23.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:50:23.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:50:23.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:50:23.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:50:23.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:50:23.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:50:23.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:50:23.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:50:23.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-16T16:50:23.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:50:23.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=225bc5d4-c397-4ef2-9ccc-e28499dd3fa7 2024-09-16T16:50:23.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 225bc5d4-c397-4ef2-9ccc-e28499dd3fa7' 2024-09-16T16:50:23.783 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 225bc5d4-c397-4ef2-9ccc-e28499dd3fa7 2024-09-16T16:50:23.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:50:23.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPYehmGfiWLxAA3Ha8j46xGy6fQ8gSfeNC2w== 2024-09-16T16:50:23.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPYehmGfiWLxAA3Ha8j46xGy6fQ8gSfeNC2w=="}' 2024-09-16T16:50:23.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 225bc5d4-c397-4ef2-9ccc-e28499dd3fa7 -i td/osd-recovery-stats/0/new.json 2024-09-16T16:50:24.147 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:50:24.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-16T16:50:24.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPYehmGfiWLxAA3Ha8j46xGy6fQ8gSfeNC2w== --osd-uuid 225bc5d4-c397-4ef2-9ccc-e28499dd3fa7 2024-09-16T16:50:24.197 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:24.195+0000 7fcc91f747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:50:24.206 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:24.207+0000 7fcc91f747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:50:24.208 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:24.207+0000 7fcc91f747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:50:24.209 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:24.207+0000 7fcc91f747c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:50:24.209 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:24.207+0000 7fcc91f747c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:50:24.209 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:24.207+0000 7fcc91f747c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:50:24.247 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:24.247+0000 7fcc91f747c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-16T16:50:30.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-16T16:50:30.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:50:30.501 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:50:30.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:50:30.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:50:31.078 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:50:31.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:50:31.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:50:31.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:50:31.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:50:31.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:50:31.110 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:31.106+0000 7f8f41be67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:50:31.113 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:31.110+0000 7f8f41be67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:50:31.114 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:31.110+0000 7f8f41be67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:50:31.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:50:31.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:50:31.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:50:31.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:50:31.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:50:31.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:31.527 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:50:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:50:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:31.688 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:31.686+0000 7f8f41be67c0 -1 Falling back to public interface 2024-09-16T16:50:31.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:32.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:32.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:32.969 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:50:32.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:50:32.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:32.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:33.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:34.387 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:50:34.386+0000 7f8f41be67c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:50:34.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:34.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:34.391 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:50:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:50:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:34.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:35.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:35.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:35.870 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:50:35.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:50:35.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:35.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:36.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:37.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:37.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:37.385 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:50:37.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:50:37.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:37.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:37.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:38.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:38.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:38.833 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:50:38.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:50:38.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:38.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:39.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:40.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:40.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:40.381 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:50:40.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:50:40.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:40.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:40.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:41.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:41.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:41.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:50:41.818 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:50:41.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:41.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:42.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:43.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:43.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:43.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:50:43.391 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:50:43.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:43.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:43.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:44.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:44.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:44.829 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:50:44.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:50:44.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:44.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:45.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:46.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:46.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:46.318 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:50:46.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:50:46.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:46.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:46.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:47.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:47.750 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:50:47.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:47.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:50:47.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:47.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:48.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:49.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:49.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:49.184 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:50:49.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:50:49.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:49.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:50.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:51.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:51.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:51.043 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:50:51.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:50:51.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:51.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:52.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:53.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:53.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:53.168 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:50:53.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:50:53.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:53.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:54.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:55.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:55.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:55.052 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:50:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:50:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:56.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:57.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:57.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:57.122 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:50:57.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:50:57.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:57.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:57.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:50:58.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:50:58.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:50:58.860 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:50:58.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:50:58.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:50:58.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:50:59.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:00.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:00.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:00.362 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T16:51:00.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T16:51:00.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:00.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:51:00.838 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2089342768,v1:127.0.0.1:6803/2089342768] [v2:127.0.0.1:6804/2089342768,v1:127.0.0.1:6805/2089342768] exists,up 225bc5d4-c397-4ef2-9ccc-e28499dd3fa7 2024-09-16T16:51:00.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:51:00.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:51:00.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:51:00.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:191: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 1 2024-09-16T16:51:00.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:51:00.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:51:00.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:51:00.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:51:00.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-16T16:51:00.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:51:00.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:51:00.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:51:00.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:51:00.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-16T16:51:00.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-16T16:51:00.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:51:00.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:51:00.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:51:00.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:51:00.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:51:00.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:51:00.844 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:51:00.844 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:51:00.844 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:51:00.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:51:00.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:51:00.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:51:00.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:51:00.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:51:00.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:51:00.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:51:00.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:51:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:51:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:51:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:51:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:51:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:51:00.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-16T16:51:00.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:51:00.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dc342657-7649-4b3c-9b28-7c641abaefb9 2024-09-16T16:51:00.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 dc342657-7649-4b3c-9b28-7c641abaefb9' 2024-09-16T16:51:00.846 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 dc342657-7649-4b3c-9b28-7c641abaefb9 2024-09-16T16:51:00.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:51:00.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0YehmQ8llMxAA729kFpTAolnl83Zf9TdkPw== 2024-09-16T16:51:00.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0YehmQ8llMxAA729kFpTAolnl83Zf9TdkPw=="}' 2024-09-16T16:51:00.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dc342657-7649-4b3c-9b28-7c641abaefb9 -i td/osd-recovery-stats/1/new.json 2024-09-16T16:51:01.402 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:51:01.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-16T16:51:01.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0YehmQ8llMxAA729kFpTAolnl83Zf9TdkPw== --osd-uuid dc342657-7649-4b3c-9b28-7c641abaefb9 2024-09-16T16:51:01.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:01.458+0000 7f00ef1087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:01.463 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:01.462+0000 7f00ef1087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:01.465 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:01.462+0000 7f00ef1087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:01.465 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:01.462+0000 7f00ef1087c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:51:01.465 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:01.462+0000 7f00ef1087c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:51:01.466 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:01.462+0000 7f00ef1087c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:51:01.505 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:01.502+0000 7f00ef1087c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-16T16:51:08.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-16T16:51:08.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:51:08.202 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:51:08.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:51:08.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:51:08.754 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:51:08.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:51:08.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:51:08.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:51:08.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:51:08.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:51:08.787 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:08.786+0000 7fb6044097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:08.799 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:08.798+0000 7fb6044097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:08.800 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:08.798+0000 7fb6044097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:09.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:51:09.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:51:09.213 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:51:09.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:51:09.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:51:09.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:51:09.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:09.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:51:09.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:09.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:09.341 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:09.338+0000 7fb6044097c0 -1 Falling back to public interface 2024-09-16T16:51:09.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:10.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:10.656 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:51:10.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:51:10.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:10.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:11.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:11.899 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:11.898+0000 7fb6044097c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:51:12.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:12.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:12.114 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:51:12.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:51:12.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:12.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:12.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:13.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:13.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:13.643 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:51:13.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:51:13.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:13.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:14.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:15.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:15.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:15.109 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:51:15.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:51:15.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:15.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:15.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:16.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:16.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:16.645 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:51:16.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:51:16.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:16.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:17.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:18.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:18.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:18.241 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:51:18.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:51:18.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:18.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:18.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:19.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:19.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:19.748 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:51:19.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:51:19.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:19.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:20.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:21.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:21.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:21.195 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:51:21.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:51:21.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:21.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:21.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:22.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:22.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:22.764 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:51:22.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:51:22.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:22.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:23.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:24.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:24.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:24.255 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:51:24.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:51:24.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:24.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:24.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:25.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:25.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:25.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:51:25.831 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:51:25.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:25.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:26.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:27.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:27.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:27.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:51:27.260 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:51:27.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:27.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:28.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:29.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:29.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:29.227 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:51:29.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:51:29.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:29.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:30.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:31.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:31.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:31.539 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:51:31.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:51:31.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:31.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:32.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:33.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:33.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:33.366 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:51:33.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:51:33.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:33.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:34.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:35.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:35.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:35.394 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:51:35.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:51:35.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:35.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:36.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:37.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:37.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:37.628 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:51:37.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:51:37.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:37.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:51:38.116 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2652890365,v1:127.0.0.1:6811/2652890365] [v2:127.0.0.1:6812/2652890365,v1:127.0.0.1:6813/2652890365] exists,up dc342657-7649-4b3c-9b28-7c641abaefb9 2024-09-16T16:51:38.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:51:38.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:51:38.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:51:38.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:192: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 2 2024-09-16T16:51:38.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:51:38.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:51:38.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T16:51:38.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:51:38.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-16T16:51:38.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:51:38.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:51:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:51:38.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:51:38.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:51:38.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:51:38.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:51:38.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:51:38.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:51:38.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:51:38.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:51:38.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:51:38.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:51:38.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:51:38.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:51:38.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:51:38.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-16T16:51:38.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:51:38.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=83320675-96d4-42c4-94fb-b28aca9709e4 2024-09-16T16:51:38.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 83320675-96d4-42c4-94fb-b28aca9709e4' 2024-09-16T16:51:38.124 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 83320675-96d4-42c4-94fb-b28aca9709e4 2024-09-16T16:51:38.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:51:38.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaYuhm6z9GCBAA/JHv6SYvtKkzSXw4sGq5Yg== 2024-09-16T16:51:38.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaYuhm6z9GCBAA/JHv6SYvtKkzSXw4sGq5Yg=="}' 2024-09-16T16:51:38.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 83320675-96d4-42c4-94fb-b28aca9709e4 -i td/osd-recovery-stats/2/new.json 2024-09-16T16:51:38.763 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:51:38.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-16T16:51:38.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAaYuhm6z9GCBAA/JHv6SYvtKkzSXw4sGq5Yg== --osd-uuid 83320675-96d4-42c4-94fb-b28aca9709e4 2024-09-16T16:51:38.817 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:38.813+0000 7ffb44bed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:38.819 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:38.817+0000 7ffb44bed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:38.821 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:38.817+0000 7ffb44bed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:38.821 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:38.817+0000 7ffb44bed7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:51:38.821 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:38.817+0000 7ffb44bed7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:51:38.821 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:38.817+0000 7ffb44bed7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:51:38.918 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:38.917+0000 7ffb44bed7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-16T16:51:45.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-16T16:51:45.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:51:45.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T16:51:45.512 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T16:51:45.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:51:46.082 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T16:51:46.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T16:51:46.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:51:46.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:51:46.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:51:46.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:51:46.114 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:46.109+0000 7f2ea76ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:46.117 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:46.113+0000 7f2ea76ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:46.118 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:46.113+0000 7f2ea76ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:51:46.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T16:51:46.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:51:46.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T16:51:46.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:51:46.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:51:46.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:46.530 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:51:46.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:51:46.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:46.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:51:46.716 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:46.713+0000 7f2ea76ec7c0 -1 Falling back to public interface 2024-09-16T16:51:46.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:47.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:47.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:47.963 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:51:47.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:51:47.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:47.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:51:48.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:49.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:49.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:49.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:51:49.393 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:51:49.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:49.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:51:49.471 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:51:49.469+0000 7f2ea76ec7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T16:51:49.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:50.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:50.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:50.813 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:51:50.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:51:50.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:50.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:51:51.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:52.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:52.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:52.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:51:52.398 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:51:52.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:52.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:51:52.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:53.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:53.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:53.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:51:53.858 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:51:53.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:53.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:51:54.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:55.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:55.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:55.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:51:55.405 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:51:55.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:55.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:51:55.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:56.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:56.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:56.849 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:51:56.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:51:56.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:56.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:51:57.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:58.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:58.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:58.289 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:51:58.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:51:58.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:58.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:51:58.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:51:59.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:51:59.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:51:59.715 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:51:59.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:51:59.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:51:59.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:52:00.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:01.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:01.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:01.244 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:52:01.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:52:01.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:01.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:52:01.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:02.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:02.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:02.690 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:52:02.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:52:02.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:02.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:52:03.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:04.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:04.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:04.162 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:52:04.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:52:04.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:04.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:52:04.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:05.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:05.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:05.919 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:52:05.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:52:05.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:05.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:52:06.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:07.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:07.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:07.717 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:52:07.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:52:07.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:07.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:52:08.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:09.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:09.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:09.524 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:52:09.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:52:09.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:09.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:52:10.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:11.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:11.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:11.384 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:52:11.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:52:11.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:11.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:52:11.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:12.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:12.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:12.951 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:52:12.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:52:12.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:12.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:52:13.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:14.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:14.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:14.759 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T16:52:14.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T16:52:14.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:14.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:52:15.254 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/790743696,v1:127.0.0.1:6819/790743696] [v2:127.0.0.1:6820/790743696,v1:127.0.0.1:6821/790743696] exists,up 83320675-96d4-42c4-94fb-b28aca9709e4 2024-09-16T16:52:15.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:52:15.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:52:15.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:52:15.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:193: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 3 2024-09-16T16:52:15.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:52:15.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:52:15.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T16:52:15.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:52:15.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:52:15.259 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:52:15.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-16T16:52:15.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:52:15.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=37a9bc36-2c5c-4a65-bf76-81d692ac0070 2024-09-16T16:52:15.261 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 37a9bc36-2c5c-4a65-bf76-81d692ac0070 2024-09-16T16:52:15.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 37a9bc36-2c5c-4a65-bf76-81d692ac0070' 2024-09-16T16:52:15.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:52:15.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/Yuhmidq/EBAAD9pwaET7UskxAubeawYArg== 2024-09-16T16:52:15.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/Yuhmidq/EBAAD9pwaET7UskxAubeawYArg=="}' 2024-09-16T16:52:15.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 37a9bc36-2c5c-4a65-bf76-81d692ac0070 -i td/osd-recovery-stats/3/new.json 2024-09-16T16:52:15.928 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:52:15.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-16T16:52:15.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/Yuhmidq/EBAAD9pwaET7UskxAubeawYArg== --osd-uuid 37a9bc36-2c5c-4a65-bf76-81d692ac0070 2024-09-16T16:52:15.983 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:15.980+0000 7fcb460157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:52:16.008 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:16.004+0000 7fcb460157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:52:16.009 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:16.004+0000 7fcb460157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:52:16.010 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:16.004+0000 7fcb460157c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:52:16.010 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:16.004+0000 7fcb460157c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:52:16.010 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:16.004+0000 7fcb460157c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-16T16:52:16.100 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:16.096+0000 7fcb460157c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-16T16:52:22.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-16T16:52:22.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:52:22.505 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T16:52:22.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T16:52:22.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:52:23.113 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T16:52:23.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T16:52:23.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:52:23.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:52:23.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:52:23.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:52:23.139 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:23.136+0000 7f871d7477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:52:23.157 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:23.152+0000 7f871d7477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:52:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:23.156+0000 7f871d7477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:52:23.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T16:52:23.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:52:23.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T16:52:23.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:52:23.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:52:23.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:23.532 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:52:23.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:52:23.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:23.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:23.724 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:23.720+0000 7f871d7477c0 -1 Falling back to public interface 2024-09-16T16:52:23.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:24.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:24.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:24.948 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:52:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:52:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:25.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:26.309 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:26.304+0000 7f871d7477c0 -1 osd.3 0 log_to_monitors true 2024-09-16T16:52:26.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:26.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:26.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:52:26.379 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:52:26.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:26.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:27.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:28.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:28.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:28.040 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:52:28.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:52:28.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:28.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:29.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:29.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:29.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:52:29.484 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:52:29.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:29.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:30.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:31.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:31.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:31.028 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:52:31.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:52:31.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:31.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:31.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:32.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:32.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:32.522 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:52:32.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:52:32.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:32.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:33.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:34.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:34.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:34.209 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:52:34.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:52:34.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:34.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:34.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:35.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:52:35.671 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:52:35.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:35.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:36.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:37.152 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:52:37.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:52:37.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:37.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:37.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:38.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:38.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:38.618 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:52:38.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:52:38.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:38.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:40.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:40.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:40.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:52:40.163 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:52:40.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:40.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:40.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:41.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:41.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:41.907 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:52:41.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:52:41.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:41.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:42.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:43.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:43.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:43.676 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:52:43.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:52:43.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:43.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:44.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:45.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:45.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:45.484 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:52:45.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:52:45.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:45.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:46.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:47.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:47.336 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:52:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:52:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:47.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:48.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:49.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:49.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:49.435 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:52:49.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:52:49.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:49.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:50.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:52:51.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:52:51.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:52:51.439 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:52:51.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:52:51.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:52:51.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T16:52:51.896 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2721170550,v1:127.0.0.1:6827/2721170550] [v2:127.0.0.1:6828/2721170550,v1:127.0.0.1:6829/2721170550] exists,up 37a9bc36-2c5c-4a65-bf76-81d692ac0070 2024-09-16T16:52:51.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:52:51.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:194: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 4 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:52:51.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:52:51.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:52:51.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:52:51.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:52:51.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:52:51.900 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:52:51.900 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:52:51.900 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:52:51.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:52:51.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:52:51.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:52:51.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:52:51.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:52:51.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:52:51.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:52:51.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:52:51.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:52:51.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:52:51.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:52:51.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:52:51.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:52:51.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-16T16:52:51.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:52:51.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a8d00cf7-7650-451d-a4af-68a1df91bb16 2024-09-16T16:52:51.903 INFO:tasks.workunit.client.0.smithi039.stdout:add osd4 a8d00cf7-7650-451d-a4af-68a1df91bb16 2024-09-16T16:52:51.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 a8d00cf7-7650-451d-a4af-68a1df91bb16' 2024-09-16T16:52:51.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:52:51.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjYuhmKZpFNxAAugtaa2IfyqIRS8i9pz8iYA== 2024-09-16T16:52:51.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjYuhmKZpFNxAAugtaa2IfyqIRS8i9pz8iYA=="}' 2024-09-16T16:52:51.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a8d00cf7-7650-451d-a4af-68a1df91bb16 -i td/osd-recovery-stats/4/new.json 2024-09-16T16:52:52.492 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:52:52.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-16T16:52:52.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjYuhmKZpFNxAAugtaa2IfyqIRS8i9pz8iYA== --osd-uuid a8d00cf7-7650-451d-a4af-68a1df91bb16 2024-09-16T16:52:52.538 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:52.532+0000 7f45dbce67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:52:52.540 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:52.536+0000 7f45dbce67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:52:52.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:52.536+0000 7f45dbce67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:52:52.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:52.536+0000 7f45dbce67c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:52:52.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:52.536+0000 7f45dbce67c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:52:52.541 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:52.536+0000 7f45dbce67c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-16T16:52:52.630 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:52.624+0000 7f45dbce67c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-16T16:52:59.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-16T16:52:59.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:52:59.329 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd4 key to auth repository 2024-09-16T16:52:59.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T16:52:59.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:52:59.883 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.4 2024-09-16T16:52:59.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T16:52:59.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:52:59.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:52:59.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:52:59.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:52:59.917 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:59.911+0000 7f18521527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:52:59.919 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:59.915+0000 7f18521527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:52:59.921 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:52:59.915+0000 7f18521527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:53:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T16:53:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:53:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T16:53:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:53:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:53:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:00.322 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:53:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:53:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:00.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:00.468 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:00.463+0000 7f18521527c0 -1 Falling back to public interface 2024-09-16T16:53:00.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:01.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:01.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:01.866 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:53:01.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:53:01.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:01.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:02.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:03.226 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:03.219+0000 7f18521527c0 -1 osd.4 0 log_to_monitors true 2024-09-16T16:53:03.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:03.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:03.332 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:53:03.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:53:03.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:03.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:04.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:05.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:05.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:05.002 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:53:05.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:53:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:05.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:05.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:06.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:06.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:06.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:53:06.441 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:53:06.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:06.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:06.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:07.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:07.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:07.915 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:53:07.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:53:07.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:07.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:09.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:09.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:09.483 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:53:09.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:53:09.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:09.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:09.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:10.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:10.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:10.922 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:53:10.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:53:10.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:10.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:11.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:12.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:12.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:12.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:53:12.467 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:53:12.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:12.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:12.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:13.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:13.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:13.918 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:53:13.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:53:13.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:13.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:14.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:15.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:15.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:15.415 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:53:15.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:53:15.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:15.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:15.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:16.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:16.880 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:53:16.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:16.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:53:16.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:16.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:17.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:18.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:18.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:18.810 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:53:18.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:53:18.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:18.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:19.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:20.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:20.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:20.479 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:53:20.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:53:20.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:20.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:21.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:22.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:22.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:22.406 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:53:22.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:53:22.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:22.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:23.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:24.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:24.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:24.191 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:53:24.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:53:24.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:24.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:25.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:26.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:26.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:26.161 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:53:26.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:53:26.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:26.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:26.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:27.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:27.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:27.986 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:53:27.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:53:27.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:27.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:28.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:29.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:29.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:29.460 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T16:53:29.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T16:53:29.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:29.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T16:53:29.905 INFO:tasks.workunit.client.0.smithi039.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2405015228,v1:127.0.0.1:6835/2405015228] [v2:127.0.0.1:6836/2405015228,v1:127.0.0.1:6837/2405015228] exists,up a8d00cf7-7650-451d-a4af-68a1df91bb16 2024-09-16T16:53:29.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:53:29.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:53:29.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:195: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 5 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-16T16:53:29.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:53:29.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:53:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:53:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:53:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:53:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:53:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:53:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:53:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:53:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:53:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:53:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:53:29.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-16T16:53:29.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:53:29.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=45d6734e-0501-4325-b9b4-aa327f5358f9 2024-09-16T16:53:29.913 INFO:tasks.workunit.client.0.smithi039.stdout:add osd5 45d6734e-0501-4325-b9b4-aa327f5358f9 2024-09-16T16:53:29.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 45d6734e-0501-4325-b9b4-aa327f5358f9' 2024-09-16T16:53:29.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:53:29.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJYuhmA7JnNxAAro8t/w9xDd18qwLKbsgLBg== 2024-09-16T16:53:29.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJYuhmA7JnNxAAro8t/w9xDd18qwLKbsgLBg=="}' 2024-09-16T16:53:29.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 45d6734e-0501-4325-b9b4-aa327f5358f9 -i td/osd-recovery-stats/5/new.json 2024-09-16T16:53:30.468 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:53:30.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-16T16:53:30.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJYuhmA7JnNxAAro8t/w9xDd18qwLKbsgLBg== --osd-uuid 45d6734e-0501-4325-b9b4-aa327f5358f9 2024-09-16T16:53:30.516 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:30.511+0000 7f02e31e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:53:30.551 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:30.547+0000 7f02e31e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:53:30.552 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:30.547+0000 7f02e31e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:53:30.553 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:30.547+0000 7f02e31e37c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:53:30.553 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:30.547+0000 7f02e31e37c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:53:30.553 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:30.547+0000 7f02e31e37c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-16T16:53:30.590 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:30.583+0000 7f02e31e37c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-16T16:53:37.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-16T16:53:37.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:53:37.652 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd5 key to auth repository 2024-09-16T16:53:37.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T16:53:37.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:53:38.218 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.5 2024-09-16T16:53:38.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T16:53:38.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:53:38.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:53:38.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:53:38.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:53:38.251 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:38.251+0000 7f04f57677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:53:38.273 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:38.271+0000 7f04f57677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:53:38.275 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:38.275+0000 7f04f57677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:53:38.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T16:53:38.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:53:38.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T16:53:38.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:53:38.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:53:38.639 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:53:38.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:38.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:53:38.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:38.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:38.833 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:38.831+0000 7f04f57677c0 -1 Falling back to public interface 2024-09-16T16:53:39.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:40.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:40.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:40.097 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:53:40.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:53:40.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:40.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:40.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:41.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:41.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:41.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:53:41.521 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:53:41.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:41.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:41.539 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:53:41.539+0000 7f04f57677c0 -1 osd.5 0 log_to_monitors true 2024-09-16T16:53:42.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:43.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:43.039 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:53:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:53:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:43.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:43.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:44.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:44.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:44.487 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:53:44.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:53:44.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:44.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:45.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:46.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:46.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:46.011 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:53:46.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:53:46.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:46.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:46.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:47.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:47.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:47.584 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:53:47.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:53:47.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:47.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:48.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:49.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:49.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:49.053 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:53:49.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:53:49.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:49.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:49.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:50.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:50.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:50.623 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:53:50.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:53:50.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:50.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:51.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:52.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:52.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:52.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:53:52.206 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:53:52.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:52.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:52.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:53.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:53.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:53.683 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:53:53.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:53:53.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:53.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:54.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:55.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:55.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:55.163 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:53:55.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:53:55.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:55.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:56.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:57.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:57.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:57.167 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:53:57.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:53:57.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:57.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:53:58.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:53:59.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:53:59.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:53:59.509 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:53:59.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:53:59.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:53:59.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:54:00.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:54:01.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:54:01.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:54:01.552 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:54:01.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:54:01.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:54:01.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:54:02.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:54:03.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:54:03.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:54:03.518 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:54:03.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:54:03.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:54:03.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:54:04.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:54:05.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:54:05.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:54:05.085 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:54:05.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:54:05.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:54:05.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:54:05.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:54:06.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:54:06.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:54:06.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:54:06.777 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:54:06.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:54:06.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T16:54:07.242 INFO:tasks.workunit.client.0.smithi039.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/765373355,v1:127.0.0.1:6843/765373355] [v2:127.0.0.1:6844/765373355,v1:127.0.0.1:6845/765373355] exists,up 45d6734e-0501-4325-b9b4-aa327f5358f9 2024-09-16T16:54:07.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:54:07.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:54:07.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:54:07.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:197: TEST_recovery_sizeup: create_pool test 1 1 2024-09-16T16:54:07.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-16T16:54:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T16:54:08.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:54:09.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:198: TEST_recovery_sizeup: ceph osd pool set test size 2 2024-09-16T16:54:10.267 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-09-16T16:54:10.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:200: TEST_recovery_sizeup: wait_for_clean 2024-09-16T16:54:10.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:54:10.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:54:10.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:54:10.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:54:10.293 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:54:10.293 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:54:10.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:54:10.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:54:10.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:54:10.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:54:10.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:54:10.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:54:10.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:54:10.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:54:10.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:54:11.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:54:11.007 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:54:11.007 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:54:11.007 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:54:11.007 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T16:54:11.008 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T16:54:11.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:54:11.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:11.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:54:11.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836524 2024-09-16T16:54:11.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836524 2024-09-16T16:54:11.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836524' 2024-09-16T16:54:11.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:11.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:54:11.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672997 2024-09-16T16:54:11.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672997 2024-09-16T16:54:11.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836524 1-42949672997' 2024-09-16T16:54:11.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:11.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:54:11.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509473 2024-09-16T16:54:11.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509473 2024-09-16T16:54:11.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836524 1-42949672997 2-64424509473' 2024-09-16T16:54:11.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:11.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:54:11.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345940 2024-09-16T16:54:11.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345940 2024-09-16T16:54:11.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836524 1-42949672997 2-64424509473 3-85899345940' 2024-09-16T16:54:11.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:11.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:54:11.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182415 2024-09-16T16:54:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182415 2024-09-16T16:54:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836524 1-42949672997 2-64424509473 3-85899345940 4-107374182415' 2024-09-16T16:54:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:11.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T16:54:11.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018888 2024-09-16T16:54:11.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018888 2024-09-16T16:54:11.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836524 1-42949672997 2-64424509473 3-85899345940 4-107374182415 5-128849018888' 2024-09-16T16:54:11.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:11.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836524 2024-09-16T16:54:11.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:11.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:54:11.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836524 2024-09-16T16:54:11.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:11.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836524 2024-09-16T16:54:11.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-16T16:54:11.965 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836524 2024-09-16T16:54:11.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:54:12.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836522 -lt 21474836524 2024-09-16T16:54:12.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:54:13.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:54:13.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:54:13.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-16T16:54:13.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:13.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672997 2024-09-16T16:54:13.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:13.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:54:13.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:13.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672997 2024-09-16T16:54:13.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672997 2024-09-16T16:54:13.814 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672997 2024-09-16T16:54:13.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2024-09-16T16:54:13.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:54:14.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672997 -lt 42949672997 2024-09-16T16:54:14.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:14.236 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509473 2024-09-16T16:54:14.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:14.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:54:14.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509473 2024-09-16T16:54:14.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:14.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509473 2024-09-16T16:54:14.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-09-16T16:54:14.241 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509473 2024-09-16T16:54:14.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:54:14.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509473 -lt 64424509473 2024-09-16T16:54:14.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:14.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:14.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345940 2024-09-16T16:54:14.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:54:14.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345940 2024-09-16T16:54:14.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:14.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345940 2024-09-16T16:54:14.691 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345940 2024-09-16T16:54:14.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345940' 2024-09-16T16:54:14.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:54:15.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345941 -lt 85899345940 2024-09-16T16:54:15.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:15.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182415 2024-09-16T16:54:15.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:15.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:54:15.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182415 2024-09-16T16:54:15.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:15.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182415 2024-09-16T16:54:15.138 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182415 2024-09-16T16:54:15.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-09-16T16:54:15.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:54:15.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182416 -lt 107374182415 2024-09-16T16:54:15.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:15.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:15.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018888 2024-09-16T16:54:15.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T16:54:15.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018888 2024-09-16T16:54:15.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:15.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018888 2024-09-16T16:54:15.559 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018888 2024-09-16T16:54:15.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2024-09-16T16:54:15.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T16:54:16.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018889 -lt 128849018888 2024-09-16T16:54:16.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:54:16.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:54:16.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:54:16.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:54:16.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:54:16.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:54:16.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:54:16.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:54:16.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:54:16.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:54:16.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:54:17.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:54:17.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:54:17.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:54:17.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:54:17.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:54:17.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:54:17.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:54:17.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: seq 1 200 2024-09-16T16:54:17.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:17.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj1 /dev/null 2024-09-16T16:54:17.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:17.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj2 /dev/null 2024-09-16T16:54:17.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:17.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj3 /dev/null 2024-09-16T16:54:17.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:17.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj4 /dev/null 2024-09-16T16:54:18.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:18.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj5 /dev/null 2024-09-16T16:54:18.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:18.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj6 /dev/null 2024-09-16T16:54:18.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:18.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj7 /dev/null 2024-09-16T16:54:18.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:18.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj8 /dev/null 2024-09-16T16:54:18.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:18.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj9 /dev/null 2024-09-16T16:54:18.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:18.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj10 /dev/null 2024-09-16T16:54:18.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:18.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj11 /dev/null 2024-09-16T16:54:18.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:18.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj12 /dev/null 2024-09-16T16:54:19.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:19.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj13 /dev/null 2024-09-16T16:54:19.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:19.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj14 /dev/null 2024-09-16T16:54:19.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:19.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj15 /dev/null 2024-09-16T16:54:19.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:19.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj16 /dev/null 2024-09-16T16:54:19.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:19.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj17 /dev/null 2024-09-16T16:54:19.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:19.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj18 /dev/null 2024-09-16T16:54:19.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:19.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj19 /dev/null 2024-09-16T16:54:19.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:19.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj20 /dev/null 2024-09-16T16:54:19.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:19.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj21 /dev/null 2024-09-16T16:54:20.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:20.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj22 /dev/null 2024-09-16T16:54:20.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:20.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj23 /dev/null 2024-09-16T16:54:20.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:20.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj24 /dev/null 2024-09-16T16:54:20.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:20.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj25 /dev/null 2024-09-16T16:54:20.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:20.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj26 /dev/null 2024-09-16T16:54:20.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:20.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj27 /dev/null 2024-09-16T16:54:20.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:20.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj28 /dev/null 2024-09-16T16:54:20.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:20.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj29 /dev/null 2024-09-16T16:54:21.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:21.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj30 /dev/null 2024-09-16T16:54:21.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:21.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj31 /dev/null 2024-09-16T16:54:21.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:21.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj32 /dev/null 2024-09-16T16:54:21.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:21.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj33 /dev/null 2024-09-16T16:54:21.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:21.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj34 /dev/null 2024-09-16T16:54:21.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:21.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj35 /dev/null 2024-09-16T16:54:21.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:21.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj36 /dev/null 2024-09-16T16:54:21.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:21.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj37 /dev/null 2024-09-16T16:54:22.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:22.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj38 /dev/null 2024-09-16T16:54:22.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:22.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj39 /dev/null 2024-09-16T16:54:22.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:22.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj40 /dev/null 2024-09-16T16:54:22.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:22.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj41 /dev/null 2024-09-16T16:54:22.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:22.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj42 /dev/null 2024-09-16T16:54:22.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:22.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj43 /dev/null 2024-09-16T16:54:22.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:22.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj44 /dev/null 2024-09-16T16:54:22.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:22.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj45 /dev/null 2024-09-16T16:54:22.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:22.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj46 /dev/null 2024-09-16T16:54:23.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:23.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj47 /dev/null 2024-09-16T16:54:23.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:23.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj48 /dev/null 2024-09-16T16:54:23.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:23.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj49 /dev/null 2024-09-16T16:54:23.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:23.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj50 /dev/null 2024-09-16T16:54:23.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:23.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj51 /dev/null 2024-09-16T16:54:23.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:23.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj52 /dev/null 2024-09-16T16:54:23.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:23.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj53 /dev/null 2024-09-16T16:54:23.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:23.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj54 /dev/null 2024-09-16T16:54:23.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:23.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj55 /dev/null 2024-09-16T16:54:24.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:24.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj56 /dev/null 2024-09-16T16:54:24.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:24.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj57 /dev/null 2024-09-16T16:54:24.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:24.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj58 /dev/null 2024-09-16T16:54:24.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:24.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj59 /dev/null 2024-09-16T16:54:24.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:24.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj60 /dev/null 2024-09-16T16:54:24.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:24.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj61 /dev/null 2024-09-16T16:54:24.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:24.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj62 /dev/null 2024-09-16T16:54:24.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:24.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj63 /dev/null 2024-09-16T16:54:25.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:25.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj64 /dev/null 2024-09-16T16:54:25.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:25.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj65 /dev/null 2024-09-16T16:54:25.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:25.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj66 /dev/null 2024-09-16T16:54:25.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:25.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj67 /dev/null 2024-09-16T16:54:25.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:25.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj68 /dev/null 2024-09-16T16:54:25.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:25.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj69 /dev/null 2024-09-16T16:54:25.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:25.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj70 /dev/null 2024-09-16T16:54:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj71 /dev/null 2024-09-16T16:54:25.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:25.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj72 /dev/null 2024-09-16T16:54:26.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:26.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj73 /dev/null 2024-09-16T16:54:26.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:26.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj74 /dev/null 2024-09-16T16:54:26.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:26.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj75 /dev/null 2024-09-16T16:54:26.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:26.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj76 /dev/null 2024-09-16T16:54:26.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:26.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj77 /dev/null 2024-09-16T16:54:26.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:26.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj78 /dev/null 2024-09-16T16:54:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj79 /dev/null 2024-09-16T16:54:26.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:26.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj80 /dev/null 2024-09-16T16:54:27.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:27.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj81 /dev/null 2024-09-16T16:54:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj82 /dev/null 2024-09-16T16:54:27.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:27.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj83 /dev/null 2024-09-16T16:54:27.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:27.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj84 /dev/null 2024-09-16T16:54:27.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:27.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj85 /dev/null 2024-09-16T16:54:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:27.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj86 /dev/null 2024-09-16T16:54:27.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:27.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj87 /dev/null 2024-09-16T16:54:27.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:27.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj88 /dev/null 2024-09-16T16:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:27.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj89 /dev/null 2024-09-16T16:54:28.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:28.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj90 /dev/null 2024-09-16T16:54:28.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:28.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj91 /dev/null 2024-09-16T16:54:28.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:28.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj92 /dev/null 2024-09-16T16:54:28.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:28.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj93 /dev/null 2024-09-16T16:54:28.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:28.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj94 /dev/null 2024-09-16T16:54:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj95 /dev/null 2024-09-16T16:54:28.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:28.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj96 /dev/null 2024-09-16T16:54:28.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:28.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj97 /dev/null 2024-09-16T16:54:29.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:29.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj98 /dev/null 2024-09-16T16:54:29.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:29.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj99 /dev/null 2024-09-16T16:54:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:29.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj100 /dev/null 2024-09-16T16:54:29.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:29.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj101 /dev/null 2024-09-16T16:54:29.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:29.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj102 /dev/null 2024-09-16T16:54:29.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:29.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj103 /dev/null 2024-09-16T16:54:29.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:29.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj104 /dev/null 2024-09-16T16:54:29.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:29.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj105 /dev/null 2024-09-16T16:54:29.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:29.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj106 /dev/null 2024-09-16T16:54:30.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:30.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj107 /dev/null 2024-09-16T16:54:30.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:30.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj108 /dev/null 2024-09-16T16:54:30.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:30.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj109 /dev/null 2024-09-16T16:54:30.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:30.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj110 /dev/null 2024-09-16T16:54:30.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:30.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj111 /dev/null 2024-09-16T16:54:30.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:30.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj112 /dev/null 2024-09-16T16:54:30.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:30.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj113 /dev/null 2024-09-16T16:54:30.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:30.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj114 /dev/null 2024-09-16T16:54:31.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:31.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj115 /dev/null 2024-09-16T16:54:31.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:31.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj116 /dev/null 2024-09-16T16:54:31.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:31.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj117 /dev/null 2024-09-16T16:54:31.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:31.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj118 /dev/null 2024-09-16T16:54:31.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:31.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj119 /dev/null 2024-09-16T16:54:31.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:31.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj120 /dev/null 2024-09-16T16:54:31.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:31.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj121 /dev/null 2024-09-16T16:54:31.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:31.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj122 /dev/null 2024-09-16T16:54:32.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:32.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj123 /dev/null 2024-09-16T16:54:32.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:32.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj124 /dev/null 2024-09-16T16:54:32.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:32.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj125 /dev/null 2024-09-16T16:54:32.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:32.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj126 /dev/null 2024-09-16T16:54:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj127 /dev/null 2024-09-16T16:54:32.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:32.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj128 /dev/null 2024-09-16T16:54:32.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:32.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj129 /dev/null 2024-09-16T16:54:32.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:32.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj130 /dev/null 2024-09-16T16:54:32.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:32.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj131 /dev/null 2024-09-16T16:54:33.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:33.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj132 /dev/null 2024-09-16T16:54:33.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:33.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj133 /dev/null 2024-09-16T16:54:33.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:33.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj134 /dev/null 2024-09-16T16:54:33.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:33.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj135 /dev/null 2024-09-16T16:54:33.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:33.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj136 /dev/null 2024-09-16T16:54:33.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:33.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj137 /dev/null 2024-09-16T16:54:34.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:34.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj138 /dev/null 2024-09-16T16:54:34.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:34.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj139 /dev/null 2024-09-16T16:54:34.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:34.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj140 /dev/null 2024-09-16T16:54:34.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:34.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj141 /dev/null 2024-09-16T16:54:34.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:34.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj142 /dev/null 2024-09-16T16:54:34.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:34.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj143 /dev/null 2024-09-16T16:54:34.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:34.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj144 /dev/null 2024-09-16T16:54:34.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:34.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj145 /dev/null 2024-09-16T16:54:34.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:34.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj146 /dev/null 2024-09-16T16:54:35.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:35.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj147 /dev/null 2024-09-16T16:54:35.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:35.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj148 /dev/null 2024-09-16T16:54:35.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:35.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj149 /dev/null 2024-09-16T16:54:35.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:35.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj150 /dev/null 2024-09-16T16:54:35.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:35.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj151 /dev/null 2024-09-16T16:54:35.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:35.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj152 /dev/null 2024-09-16T16:54:35.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:35.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj153 /dev/null 2024-09-16T16:54:35.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:35.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj154 /dev/null 2024-09-16T16:54:35.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:35.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj155 /dev/null 2024-09-16T16:54:36.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:36.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj156 /dev/null 2024-09-16T16:54:36.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:36.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj157 /dev/null 2024-09-16T16:54:36.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:36.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj158 /dev/null 2024-09-16T16:54:36.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:36.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj159 /dev/null 2024-09-16T16:54:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj160 /dev/null 2024-09-16T16:54:36.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:36.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj161 /dev/null 2024-09-16T16:54:36.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:36.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj162 /dev/null 2024-09-16T16:54:36.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:36.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj163 /dev/null 2024-09-16T16:54:37.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:37.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj164 /dev/null 2024-09-16T16:54:37.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:37.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj165 /dev/null 2024-09-16T16:54:37.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:37.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj166 /dev/null 2024-09-16T16:54:37.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:37.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj167 /dev/null 2024-09-16T16:54:37.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:37.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj168 /dev/null 2024-09-16T16:54:37.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:37.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj169 /dev/null 2024-09-16T16:54:37.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:37.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj170 /dev/null 2024-09-16T16:54:37.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:37.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj171 /dev/null 2024-09-16T16:54:37.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:37.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj172 /dev/null 2024-09-16T16:54:38.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:38.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj173 /dev/null 2024-09-16T16:54:38.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:38.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj174 /dev/null 2024-09-16T16:54:38.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:38.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj175 /dev/null 2024-09-16T16:54:38.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:38.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj176 /dev/null 2024-09-16T16:54:38.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:38.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj177 /dev/null 2024-09-16T16:54:38.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:38.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj178 /dev/null 2024-09-16T16:54:38.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:38.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj179 /dev/null 2024-09-16T16:54:38.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:38.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj180 /dev/null 2024-09-16T16:54:39.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:39.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj181 /dev/null 2024-09-16T16:54:39.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:39.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj182 /dev/null 2024-09-16T16:54:39.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:39.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj183 /dev/null 2024-09-16T16:54:39.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:39.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj184 /dev/null 2024-09-16T16:54:39.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:39.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj185 /dev/null 2024-09-16T16:54:39.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:39.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj186 /dev/null 2024-09-16T16:54:39.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:39.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj187 /dev/null 2024-09-16T16:54:39.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:39.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj188 /dev/null 2024-09-16T16:54:40.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:40.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj189 /dev/null 2024-09-16T16:54:40.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:40.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj190 /dev/null 2024-09-16T16:54:40.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:40.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj191 /dev/null 2024-09-16T16:54:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:40.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj192 /dev/null 2024-09-16T16:54:40.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:40.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj193 /dev/null 2024-09-16T16:54:40.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:40.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj194 /dev/null 2024-09-16T16:54:40.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:40.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj195 /dev/null 2024-09-16T16:54:40.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:40.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj196 /dev/null 2024-09-16T16:54:40.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:40.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj197 /dev/null 2024-09-16T16:54:41.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:41.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj198 /dev/null 2024-09-16T16:54:41.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:41.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj199 /dev/null 2024-09-16T16:54:41.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-16T16:54:41.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj200 /dev/null 2024-09-16T16:54:41.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: get_primary test obj1 2024-09-16T16:54:41.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:54:41.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:54:41.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:54:41.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:54:41.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: local primary=1 2024-09-16T16:54:41.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: get_pg test obj1 2024-09-16T16:54:41.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-16T16:54:41.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-16T16:54:41.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-16T16:54:41.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T16:54:42.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: local PG=1.0 2024-09-16T16:54:42.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: get_not_primary test obj1 2024-09-16T16:54:42.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-16T16:54:42.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-16T16:54:42.327 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-16T16:54:42.327 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:54:42.327 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:54:42.328 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:54:42.328 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:54:42.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-16T16:54:42.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-16T16:54:42.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-16T16:54:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: local otherosd=0 2024-09-16T16:54:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:212: TEST_recovery_sizeup: ceph osd set norecover 2024-09-16T16:54:44.093 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is set 2024-09-16T16:54:44.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:213: TEST_recovery_sizeup: ceph osd out osd.1 osd.0 2024-09-16T16:54:44.735 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already out. osd.0 is already out. 2024-09-16T16:54:44.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:214: TEST_recovery_sizeup: ceph osd pool set test size 4 2024-09-16T16:54:46.365 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 4 2024-09-16T16:54:46.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:215: TEST_recovery_sizeup: ceph osd unset norecover 2024-09-16T16:54:48.078 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is unset 2024-09-16T16:54:48.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: get_primary test obj1 2024-09-16T16:54:48.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:54:48.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:54:48.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:54:48.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:54:48.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: primary=2 2024-09-16T16:54:48.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:219: TEST_recovery_sizeup: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-16T16:54:48.911 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T16:54:48.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:220: TEST_recovery_sizeup: sleep 2 2024-09-16T16:54:50.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:222: TEST_recovery_sizeup: wait_for_clean 2024-09-16T16:54:50.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:54:50.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:54:50.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:54:50.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:54:50.931 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:54:50.931 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:54:50.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:54:50.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:54:50.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:54:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:54:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:54:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:54:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:54:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:54:51.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:54:51.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:54:51.491 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:54:51.491 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T16:54:51.491 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T16:54:51.491 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T16:54:51.491 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T16:54:51.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:54:51.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:51.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:54:51.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836534 2024-09-16T16:54:51.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836534 2024-09-16T16:54:51.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836534' 2024-09-16T16:54:51.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:51.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:54:51.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673007 2024-09-16T16:54:51.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673007 2024-09-16T16:54:51.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836534 1-42949673007' 2024-09-16T16:54:51.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:51.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:54:52.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509483 2024-09-16T16:54:52.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509483 2024-09-16T16:54:52.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836534 1-42949673007 2-64424509483' 2024-09-16T16:54:52.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:52.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T16:54:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345951 2024-09-16T16:54:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345951 2024-09-16T16:54:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836534 1-42949673007 2-64424509483 3-85899345951' 2024-09-16T16:54:52.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:52.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T16:54:52.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182425 2024-09-16T16:54:52.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182425 2024-09-16T16:54:52.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836534 1-42949673007 2-64424509483 3-85899345951 4-107374182425' 2024-09-16T16:54:52.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:54:52.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T16:54:52.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018899 2024-09-16T16:54:52.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018899 2024-09-16T16:54:52.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836534 1-42949673007 2-64424509483 3-85899345951 4-107374182425 5-128849018899' 2024-09-16T16:54:52.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:52.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836534 2024-09-16T16:54:52.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:52.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:54:52.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836534 2024-09-16T16:54:52.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:52.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836534 2024-09-16T16:54:52.532 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836534 2024-09-16T16:54:52.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-09-16T16:54:52.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:54:53.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836534 -lt 21474836534 2024-09-16T16:54:53.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:53.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673007 2024-09-16T16:54:53.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:53.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:54:53.013 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673007 2024-09-16T16:54:53.013 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:53.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673007 2024-09-16T16:54:53.014 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673007 2024-09-16T16:54:53.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2024-09-16T16:54:53.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:54:53.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673007 -lt 42949673007 2024-09-16T16:54:53.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:53.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:53.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509483 2024-09-16T16:54:53.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:54:53.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509483 2024-09-16T16:54:53.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:53.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509483 2024-09-16T16:54:53.566 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509483 2024-09-16T16:54:53.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509483' 2024-09-16T16:54:53.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:54:54.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509483 -lt 64424509483 2024-09-16T16:54:54.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:54.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345951 2024-09-16T16:54:54.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:54.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T16:54:54.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345951 2024-09-16T16:54:54.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:54.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345951 2024-09-16T16:54:54.047 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345951 2024-09-16T16:54:54.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2024-09-16T16:54:54.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T16:54:54.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345951 -lt 85899345951 2024-09-16T16:54:54.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:54.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182425 2024-09-16T16:54:54.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:54.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T16:54:54.525 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182425 2024-09-16T16:54:54.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:54.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182425 2024-09-16T16:54:54.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182425' 2024-09-16T16:54:54.528 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182425 2024-09-16T16:54:54.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T16:54:55.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182426 -lt 107374182425 2024-09-16T16:54:55.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:54:55.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018899 2024-09-16T16:54:55.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:54:55.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T16:54:55.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018899 2024-09-16T16:54:55.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:54:55.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018899 2024-09-16T16:54:55.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018899' 2024-09-16T16:54:55.084 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018899 2024-09-16T16:54:55.084 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T16:54:55.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018899 -lt 128849018899 2024-09-16T16:54:55.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:54:55.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:54:55.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:54:56.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:54:56.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:54:56.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:54:56.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:54:56.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:54:56.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:54:56.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:54:56.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:54:56.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:54:56.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:54:56.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:54:56.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:54:57.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:54:57.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T16:54:57.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T16:54:57.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T16:54:57.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:54:57.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:54:57.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:54:57.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:54:57.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:54:57.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:54:57.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:54:57.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:54:57.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:54:57.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:54:57.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:54:57.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:54:57.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:54:57.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:54:58.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:54:58.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:54:58.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:54:58.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:54:58.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:54:58.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:54:58.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:54:58.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:54:58.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:54:59.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-09-16T16:54:59.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-09-16T16:54:59.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:54:59.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:54:59.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:54:59.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:54:59.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:54:59.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:54:59.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:54:59.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:54:59.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:54:59.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:54:59.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:54:59.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:54:59.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:54:59.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:54:59.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:00.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:00.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:00.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:00.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:00.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:00.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:00.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:00.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:00.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:01.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-09-16T16:55:01.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-09-16T16:55:01.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:01.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:01.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:01.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:01.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:01.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:01.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:01.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:01.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:01.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:01.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:01.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:01.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:01.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:01.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:02.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:02.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:02.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:02.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:02.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:02.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:02.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:02.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:02.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:03.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-16T16:55:03.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-16T16:55:03.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:03.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:03.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:03.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:03.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:03.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:03.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:03.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:03.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:03.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:03.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:03.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:03.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:03.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:03.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:04.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:04.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:04.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:04.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:04.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:04.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:04.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:04.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:04.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:05.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-16T16:55:05.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-16T16:55:05.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:05.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:05.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:05.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:05.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:05.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:05.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:05.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:05.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:05.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:05.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:05.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:05.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:05.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:05.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:06.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:06.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:06.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:06.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:06.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:06.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:06.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:06.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:06.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:06.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-16T16:55:06.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-16T16:55:06.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:06.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:06.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:06.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:06.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:06.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:06.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:06.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:06.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:06.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:06.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:07.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:07.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:07.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:07.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:08.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:08.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:08.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:08.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:08.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:08.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:08.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:08.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:08.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:08.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-16T16:55:08.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-16T16:55:08.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:08.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:08.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:08.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:08.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:08.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:08.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:08.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:08.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:08.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:08.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:09.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:09.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:09.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:09.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:09.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:09.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:09.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:09.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:09.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:09.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:09.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:09.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:09.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:10.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-16T16:55:10.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-16T16:55:10.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:10.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:10.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:10.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:10.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:10.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:10.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:10.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:10.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:10.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:10.616 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:11.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:11.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:11.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:11.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:11.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:11.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:11.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:11.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:11.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:11.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:11.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:11.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:11.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:12.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-16T16:55:12.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-16T16:55:12.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:12.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:12.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:12.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:12.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:12.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:12.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:12.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:12.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:12.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:12.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:12.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:12.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:12.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:12.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:13.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:13.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:13.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:13.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:13.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:13.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:13.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:13.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:13.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:14.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-16T16:55:14.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-16T16:55:14.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:14.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:14.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:14.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:14.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:14.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:14.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:14.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:14.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:14.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:14.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:14.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:14.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:14.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:14.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:15.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:15.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:15.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:15.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:15.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:16.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-16T16:55:16.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-16T16:55:16.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:16.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:16.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:16.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:16.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:16.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:16.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:16.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:16.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:16.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:16.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:16.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:16.701 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:16.701 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:16.701 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:17.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:17.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:17.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:17.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:17.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:17.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:17.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:17.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:17.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:17.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-16T16:55:17.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-16T16:55:17.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:17.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:17.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:18.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:18.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:18.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:18.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:18.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:18.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:18.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:18.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:18.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:18.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:18.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:18.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:19.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:19.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:19.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:19.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:19.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:19.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:19.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:19.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:19.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:19.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-16T16:55:19.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-16T16:55:19.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:19.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:19.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:19.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:19.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:19.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:19.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:19.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:19.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:19.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:19.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:20.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:20.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:20.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:20.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:20.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:20.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:20.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:20.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:20.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:20.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:20.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:20.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:20.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:21.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-16T16:55:21.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-16T16:55:21.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:21.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:21.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:21.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:21.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:21.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:21.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:21.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:21.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:21.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:21.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:22.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:22.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:22.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:22.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:22.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:22.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:22.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:22.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:22.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:22.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:22.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:22.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:22.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:23.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-16T16:55:23.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-16T16:55:23.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:23.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:23.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:23.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:23.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:23.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:23.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:23.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:23.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:23.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:24.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:24.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:24.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:24.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:24.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:24.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:24.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:24.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:24.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:24.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:24.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:24.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:24.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:25.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-16T16:55:25.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-16T16:55:25.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:25.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:25.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:25.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:25.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:25.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:25.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:25.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:25.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:25.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:25.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:26.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T16:55:26.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:26.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:26.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:26.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T16:55:26.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T16:55:26.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T16:55:26.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T16:55:26.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T16:55:26.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T16:55:26.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T16:55:26.698 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T16:55:26.698 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T16:55:27.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-16T16:55:27.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-16T16:55:27.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-16T16:55:27.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T16:55:27.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T16:55:27.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T16:55:27.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:55:27.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:55:27.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:55:27.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:55:27.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:55:27.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:55:27.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:55:27.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:55:27.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:55:27.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:55:27.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:55:28.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:55:28.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:55:28.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:55:28.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: expr 200 '*' 2 2024-09-16T16:55:28.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: local degraded=400 2024-09-16T16:55:28.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: expr 200 '*' 2 2024-09-16T16:55:28.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: local misplaced=400 2024-09-16T16:55:28.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:226: TEST_recovery_sizeup: local log=td/osd-recovery-stats/osd.2.log 2024-09-16T16:55:28.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:227: TEST_recovery_sizeup: check td/osd-recovery-stats 1.0 2 replicated 400 0 400 0 200 0 2024-09-16T16:55:28.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-16T16:55:28.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-16T16:55:28.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=2 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=400 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=200 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.2.log 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.2.log 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-16T16:55:28.308 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-16T16:55:28.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[2,4,3,5]/[2,3]' 2024-09-16T16:55:28.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-16T16:55:28.333 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:55:28.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-09-16T16:55:28.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-09-16T16:55:28.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-16T16:55:28.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-16T16:55:28.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:55:28.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=400 2024-09-16T16:55:28.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 400 400 2024-09-16T16:55:28.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-09-16T16:55:28.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:55:28.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-09-16T16:55:28.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:55:28.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:55:28.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-09-16T16:55:28.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-16T16:55:28.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:55:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-16T16:55:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-16T16:55:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:55:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:55:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:55:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:55:28.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-16T16:55:28.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:55:28.351 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-09-16T16:55:28.351 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-09-16T16:55:28.352 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-16T16:55:28.352 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-16T16:55:28.352 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:55:28.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-09-16T16:55:28.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-09-16T16:55:28.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-09-16T16:55:28.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:55:28.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-09-16T16:55:28.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:55:28.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:55:28.355 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-09-16T16:55:28.355 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-16T16:55:28.355 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 200 ']' 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 2' 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-09-16T16:55:28.366 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-09-16T16:55:28.367 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-09-16T16:55:28.367 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:55:28.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=200 2024-09-16T16:55:28.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 200 200 2024-09-16T16:55:28.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-16T16:55:28.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:55:28.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-16T16:55:28.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:55:28.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:55:28.370 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-09-16T16:55:28.370 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-09-16T16:55:28.370 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:55:28.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-09-16T16:55:28.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-09-16T16:55:28.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:55:28.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:55:28.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:55:28.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:55:28.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:229: TEST_recovery_sizeup: delete_pool test 2024-09-16T16:55:28.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T16:55:28.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T16:55:29.065 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T16:55:29.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:230: TEST_recovery_sizeup: kill_daemons td/osd-recovery-stats 2024-09-16T16:55:29.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:55:29.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:55:29.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:55:29.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:55:29.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:55:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:55:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-16T16:55:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:55:34.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:55:34.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:55:34.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:55:34.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:55:34.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:55:34.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:55:34.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:55:34.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:55:34.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:55:34.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:55:34.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:55:34.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:55:34.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:55:34.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:55:34.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:55:34.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:55:34.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:55:34.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:55:34.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:55:34.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:55:34.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:55:34.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:55:34.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:55:34.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:55:34.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:55:34.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:55:34.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:55:34.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-16T16:55:34.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-16T16:55:34.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-16T16:55:34.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-16T16:55:34.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:55:34.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:55:34.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:55:34.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:55:34.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:55:34.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:55:34.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:55:34.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:55:34.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:55:34.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:55:34.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:55:34.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:55:34.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:55:34.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:55:34.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:55:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:55:34.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:55:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:55:34.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:55:34.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:55:34.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:55:34.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:55:34.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:55:34.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:55:34.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:55:34.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:55:34.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:55:34.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:55:34.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-16T16:55:34.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:55:34.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:55:34.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:55:34.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274941 2024-09-16T16:55:34.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:55:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T16:55:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:55:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-16T16:55:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_undersized td/osd-recovery-stats 2024-09-16T16:55:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:306: TEST_recovery_undersized: local dir=td/osd-recovery-stats 2024-09-16T16:55:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:308: TEST_recovery_undersized: local osds=3 2024-09-16T16:55:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:309: TEST_recovery_undersized: run_mon td/osd-recovery-stats a 2024-09-16T16:55:34.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-16T16:55:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:55:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:55:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:55:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-16T16:55:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-16T16:55:34.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:55:34.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:55:34.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:55:34.878 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:55:34.878 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:55:34.878 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:55:34.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:55:34.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T16:55:35.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:55:35.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:55:35.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:55:35.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:55:35.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:55:35.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:55:35.192 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:55:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:55:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:55:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:55:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:55:35.193 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:55:35.194 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:55:35.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:55:35.194 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get fsid 2024-09-16T16:55:35.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:55:35.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:55:35.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:55:35.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:55:35.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:55:35.316 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:55:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:55:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:55:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:55:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:55:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:55:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274941/ceph-mon.a.asok 2024-09-16T16:55:35.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:55:35.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274941/ceph-mon.a.asok config get mon_host 2024-09-16T16:55:35.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:310: TEST_recovery_undersized: run_mgr td/osd-recovery-stats x 2024-09-16T16:55:35.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-16T16:55:35.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:55:35.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:55:35.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:55:35.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-16T16:55:35.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:55:36.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:55:36.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:55:36.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:55:36.373 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:55:36.373 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:55:36.373 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:55:36.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:55:36.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:55:36.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:55:36.406 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: expr 3 - 1 2024-09-16T16:55:36.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: seq 0 2 2024-09-16T16:55:36.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-16T16:55:36.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 0 2024-09-16T16:55:36.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:55:36.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:55:36.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:55:36.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:55:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:55:36.412 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:55:36.412 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:55:36.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:55:36.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:55:36.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:55:36.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:55:36.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:55:36.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:55:36.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:55:36.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:55:36.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:55:36.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:55:36.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:55:36.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:55:36.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:55:36.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-16T16:55:36.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:55:36.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=06b064ac-094b-4269-a145-86b4d7a335e3 2024-09-16T16:55:36.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 06b064ac-094b-4269-a145-86b4d7a335e3' 2024-09-16T16:55:36.414 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 06b064ac-094b-4269-a145-86b4d7a335e3 2024-09-16T16:55:36.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:55:36.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIY+hmtpaJGRAA9vq0+tzsbI73itTURjD9IA== 2024-09-16T16:55:36.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIY+hmtpaJGRAA9vq0+tzsbI73itTURjD9IA=="}' 2024-09-16T16:55:36.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 06b064ac-094b-4269-a145-86b4d7a335e3 -i td/osd-recovery-stats/0/new.json 2024-09-16T16:55:36.795 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:55:36.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-16T16:55:36.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIY+hmtpaJGRAA9vq0+tzsbI73itTURjD9IA== --osd-uuid 06b064ac-094b-4269-a145-86b4d7a335e3 2024-09-16T16:55:36.852 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:36.848+0000 7f3842bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:55:36.877 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:36.872+0000 7f3842bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:55:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:36.876+0000 7f3842bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:55:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:36.876+0000 7f3842bc47c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:55:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:36.876+0000 7f3842bc47c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:55:36.879 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:36.876+0000 7f3842bc47c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-16T16:55:36.960 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:36.956+0000 7f3842bc47c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-16T16:55:43.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-16T16:55:43.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:55:43.219 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:55:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:55:43.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:55:43.853 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:55:43.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:55:43.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:55:43.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:55:43.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:55:43.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:55:43.886 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:43.884+0000 7f64bc4ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:55:43.889 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:43.884+0000 7f64bc4ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:55:43.891 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:43.888+0000 7f64bc4ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:55:44.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:55:44.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:55:44.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:55:44.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:55:44.304 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:55:44.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:55:44.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:55:44.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:55:44.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:55:44.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:55:44.452 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:44.448+0000 7f64bc4ba7c0 -1 Falling back to public interface 2024-09-16T16:55:44.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:55:45.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:55:45.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:55:45.735 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:55:45.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:55:45.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:55:45.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:55:46.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:55:47.106 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:55:47.100+0000 7f64bc4ba7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:55:47.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:55:47.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:55:47.214 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:55:47.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:55:47.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:55:47.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:55:47.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:55:48.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:55:48.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:55:48.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:55:48.771 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:55:48.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:55:48.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:55:49.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:55:50.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:55:50.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:55:50.245 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:55:50.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:55:50.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:55:50.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:55:50.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:55:51.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:55:51.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:55:51.706 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:55:51.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:55:51.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:55:51.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:55:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:55:53.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:55:53.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:55:53.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:55:53.210 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:55:53.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:55:53.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:55:53.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:55:54.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:55:54.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:55:54.675 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:55:54.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:55:54.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:55:54.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:55:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:55:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:55:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:55:56.226 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:55:56.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:55:56.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:55:56.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:55:56.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:55:57.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:55:57.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:55:57.682 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:55:57.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:55:57.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:55:57.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:55:58.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:55:59.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:55:59.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:55:59.235 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:55:59.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:55:59.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:55:59.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:55:59.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:00.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:00.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:00.904 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:56:00.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:56:00.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:00.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:56:02.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:03.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:03.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:03.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:56:03.173 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:56:03.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:03.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:56:04.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:05.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:05.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:05.133 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:56:05.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:56:05.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:05.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:56:06.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:07.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:07.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:07.677 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:56:07.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:56:07.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:07.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:56:08.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:09.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:09.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:09.256 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:56:09.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:56:09.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:09.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:56:09.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:10.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:10.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:10.735 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:56:10.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:56:10.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:10.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:56:11.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:12.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:12.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:12.352 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:56:12.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:56:12.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:12.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:56:12.779 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2984129269,v1:127.0.0.1:6803/2984129269] [v2:127.0.0.1:6804/2984129269,v1:127.0.0.1:6805/2984129269] exists,up 06b064ac-094b-4269-a145-86b4d7a335e3 2024-09-16T16:56:12.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:56:12.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 1 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-16T16:56:12.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-16T16:56:12.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:56:12.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:56:12.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:56:12.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:56:12.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:56:12.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:56:12.783 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:56:12.783 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:56:12.783 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:56:12.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:56:12.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-16T16:56:12.786 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:56:12.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9a0f6e76-7b23-4ed6-a487-245414e6f0ff 2024-09-16T16:56:12.786 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 9a0f6e76-7b23-4ed6-a487-245414e6f0ff 2024-09-16T16:56:12.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9a0f6e76-7b23-4ed6-a487-245414e6f0ff' 2024-09-16T16:56:12.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:56:12.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsY+hm7EshMBAA4th5carxSDWKn5YvBZ3Fpg== 2024-09-16T16:56:12.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsY+hm7EshMBAA4th5carxSDWKn5YvBZ3Fpg=="}' 2024-09-16T16:56:12.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9a0f6e76-7b23-4ed6-a487-245414e6f0ff -i td/osd-recovery-stats/1/new.json 2024-09-16T16:56:13.414 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:56:13.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-16T16:56:13.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsY+hm7EshMBAA4th5carxSDWKn5YvBZ3Fpg== --osd-uuid 9a0f6e76-7b23-4ed6-a487-245414e6f0ff 2024-09-16T16:56:13.473 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:13.468+0000 7faecc4ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:13.475 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:13.472+0000 7faecc4ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:13.477 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:13.472+0000 7faecc4ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:13.477 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:13.472+0000 7faecc4ff7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:56:13.477 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:13.472+0000 7faecc4ff7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:56:13.477 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:13.472+0000 7faecc4ff7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-16T16:56:13.556 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:13.552+0000 7faecc4ff7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-16T16:56:19.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-16T16:56:19.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:56:19.694 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:56:19.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:56:19.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:56:20.264 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:56:20.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:56:20.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:56:20.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:56:20.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:56:20.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:56:20.296 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:20.292+0000 7fa3ed39f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:20.337 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:20.332+0000 7fa3ed39f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:20.338 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:20.336+0000 7fa3ed39f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:20.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:56:20.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:56:20.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:56:20.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:56:20.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:56:20.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:20.729 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:56:20.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:56:20.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:20.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:20.912 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:20.908+0000 7fa3ed39f7c0 -1 Falling back to public interface 2024-09-16T16:56:21.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:22.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:22.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:22.151 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:56:22.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:56:22.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:22.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:22.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:23.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:23.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:23.581 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:56:23.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:56:23.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:23.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:23.688 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:23.683+0000 7fa3ed39f7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:56:24.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:25.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:25.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:25.014 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:56:25.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:56:25.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:25.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:25.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:26.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:26.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:26.623 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:56:26.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:56:26.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:26.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:27.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:28.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:28.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:28.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:56:28.081 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:56:28.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:28.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:28.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:29.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:29.660 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:56:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:56:29.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:29.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:30.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:31.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:31.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:31.126 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:56:31.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:56:31.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:31.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:31.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:32.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:32.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:32.774 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:56:32.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:56:32.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:32.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:33.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:34.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:34.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:34.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:56:34.236 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:56:34.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:34.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:34.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:35.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:35.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:35.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:56:35.819 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:56:35.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:35.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:36.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:37.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:37.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:37.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:56:37.272 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:56:37.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:37.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:38.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:39.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:39.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:39.067 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:56:39.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:56:39.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:39.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:40.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:41.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:41.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:41.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:56:41.082 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:56:41.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:41.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:42.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:43.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:43.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:43.050 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:56:43.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:56:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:43.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:44.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:45.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:45.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:45.265 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:56:45.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:56:45.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:45.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:46.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:47.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:47.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:47.195 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:56:47.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:56:47.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:47.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:48.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:49.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:49.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:49.341 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:56:49.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:56:49.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:49.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:50.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:56:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:56:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T16:56:51.060 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T16:56:51.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:51.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:56:51.592 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3521536665,v1:127.0.0.1:6811/3521536665] [v2:127.0.0.1:6812/3521536665,v1:127.0.0.1:6813/3521536665] exists,up 9a0f6e76-7b23-4ed6-a487-245414e6f0ff 2024-09-16T16:56:51.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:56:51.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 2 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:56:51.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:56:51.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:56:51.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:56:51.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-16T16:56:51.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-16T16:56:51.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:56:51.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:56:51.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:56:51.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:56:51.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:56:51.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:56:51.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-16T16:56:51.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:56:51.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ea05062f-7a03-45f0-ae19-5b52162ada05 2024-09-16T16:56:51.599 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 ea05062f-7a03-45f0-ae19-5b52162ada05 2024-09-16T16:56:51.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ea05062f-7a03-45f0-ae19-5b52162ada05' 2024-09-16T16:56:51.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:56:51.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTY+hmdWoIJRAA5dNifWEYbrfXCn/yPFCJ1A== 2024-09-16T16:56:51.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTY+hmdWoIJRAA5dNifWEYbrfXCn/yPFCJ1A=="}' 2024-09-16T16:56:51.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ea05062f-7a03-45f0-ae19-5b52162ada05 -i td/osd-recovery-stats/2/new.json 2024-09-16T16:56:52.210 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:56:52.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-16T16:56:52.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTY+hmdWoIJRAA5dNifWEYbrfXCn/yPFCJ1A== --osd-uuid ea05062f-7a03-45f0-ae19-5b52162ada05 2024-09-16T16:56:52.261 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:52.255+0000 7fdff32d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:52.264 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:52.259+0000 7fdff32d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:52.265 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:52.259+0000 7fdff32d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:52.265 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:52.259+0000 7fdff32d07c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:56:52.266 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:52.259+0000 7fdff32d07c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:56:52.266 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:52.259+0000 7fdff32d07c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-16T16:56:52.356 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:52.355+0000 7fdff32d07c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-16T16:56:58.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-16T16:56:58.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:56:58.853 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T16:56:58.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T16:56:58.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:56:59.435 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T16:56:59.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T16:56:59.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5a714e65-fe0d-48a4-9443-f682cd530b58 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274941/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:56:59.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:56:59.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:56:59.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:56:59.467 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:59.463+0000 7fb3dd4097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:59.470 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:59.463+0000 7fb3dd4097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:59.471 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:56:59.467+0000 7fb3dd4097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:56:59.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T16:56:59.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:56:59.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T16:56:59.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:56:59.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:56:59.875 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:56:59.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:56:59.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:56:59.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:56:59.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:00.044 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:57:00.039+0000 7fb3dd4097c0 -1 Falling back to public interface 2024-09-16T16:57:00.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:01.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:01.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:01.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:57:01.307 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:57:01.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:01.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:01.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:02.639 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:57:02.635+0000 7fb3dd4097c0 -1 osd.2 0 log_to_monitors true 2024-09-16T16:57:02.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:02.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:02.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:57:02.738 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:57:02.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:02.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:03.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:04.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:04.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:04.554 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:57:04.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:57:04.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:04.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:04.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:06.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:06.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:06.000 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:57:06.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:57:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:06.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:07.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:07.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:07.605 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:57:07.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:57:07.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:07.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:08.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:09.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:09.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:09.281 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:57:09.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:57:09.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:09.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:09.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:10.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:10.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:10.847 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:57:10.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:57:10.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:10.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:11.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:12.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:12.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:12.347 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:57:12.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:57:12.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:12.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:12.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:13.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:13.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:13.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:57:13.872 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:57:13.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:13.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:14.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:15.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:15.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:15.396 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:57:15.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:57:15.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:15.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:15.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:16.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:16.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:16.851 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:57:16.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:57:16.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:16.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:17.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:18.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:18.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:18.367 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:57:18.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:57:18.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:18.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:19.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:20.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:20.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:20.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:57:20.014 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:57:20.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:20.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:20.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:21.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:21.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:21.729 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:57:21.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:57:21.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:21.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:22.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:23.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:23.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:23.733 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:57:23.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:57:23.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:23.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:24.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:25.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:25.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:25.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:57:25.509 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:57:25.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:25.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:26.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:57:27.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:57:27.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:57:27.294 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:57:27.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:57:27.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:57:27.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:57:27.807 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/306877333,v1:127.0.0.1:6819/306877333] [v2:127.0.0.1:6820/306877333,v1:127.0.0.1:6821/306877333] exists,up ea05062f-7a03-45f0-ae19-5b52162ada05 2024-09-16T16:57:27.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:57:27.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:57:27.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:57:27.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:316: TEST_recovery_undersized: create_pool test 1 1 2024-09-16T16:57:27.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-16T16:57:28.345 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T16:57:28.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T16:57:29.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:317: TEST_recovery_undersized: ceph osd pool set test size 1 --yes-i-really-mean-it 2024-09-16T16:57:30.516 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 1 2024-09-16T16:57:30.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:319: TEST_recovery_undersized: wait_for_clean 2024-09-16T16:57:30.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T16:57:30.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T16:57:30.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T16:57:30.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T16:57:30.543 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T16:57:30.543 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T16:57:30.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T16:57:30.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T16:57:30.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T16:57:30.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T16:57:30.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T16:57:30.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T16:57:30.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T16:57:30.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:57:30.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:57:31.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:57:31.176 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:57:31.176 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T16:57:31.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:57:31.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:57:31.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:57:31.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836500 2024-09-16T16:57:31.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836500 2024-09-16T16:57:31.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500' 2024-09-16T16:57:31.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:57:31.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:57:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672975 2024-09-16T16:57:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672975 2024-09-16T16:57:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500 1-42949672975' 2024-09-16T16:57:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:57:31.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:57:31.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509447 2024-09-16T16:57:31.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509447 2024-09-16T16:57:31.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500 1-42949672975 2-64424509447' 2024-09-16T16:57:31.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:57:31.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:57:31.671 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836500 2024-09-16T16:57:31.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:57:31.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:57:31.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836500 2024-09-16T16:57:31.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836500 2024-09-16T16:57:31.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-16T16:57:31.674 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836500 2024-09-16T16:57:31.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:57:32.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-16T16:57:32.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:57:33.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:57:33.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:57:33.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836501 -lt 21474836500 2024-09-16T16:57:33.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:57:33.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672975 2024-09-16T16:57:33.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:57:33.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:57:33.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672975 2024-09-16T16:57:33.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:57:33.544 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672975 2024-09-16T16:57:33.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672975 2024-09-16T16:57:33.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-16T16:57:33.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:57:33.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-16T16:57:33.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:57:33.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509447 2024-09-16T16:57:33.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:57:33.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:57:33.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509447 2024-09-16T16:57:33.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:57:33.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509447 2024-09-16T16:57:33.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-16T16:57:33.993 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509447 2024-09-16T16:57:33.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:57:34.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-16T16:57:34.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T16:57:34.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:57:34.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:57:34.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T16:57:34.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T16:57:34.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T16:57:34.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T16:57:34.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T16:57:34.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T16:57:34.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T16:57:34.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T16:57:35.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T16:57:35.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T16:57:35.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T16:57:35.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T16:57:35.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T16:57:35.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T16:57:35.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T16:57:35.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: seq 1 200 2024-09-16T16:57:35.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:35.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj1 /dev/null 2024-09-16T16:57:36.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj2 /dev/null 2024-09-16T16:57:36.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj3 /dev/null 2024-09-16T16:57:36.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj4 /dev/null 2024-09-16T16:57:36.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj5 /dev/null 2024-09-16T16:57:36.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj6 /dev/null 2024-09-16T16:57:36.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj7 /dev/null 2024-09-16T16:57:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj8 /dev/null 2024-09-16T16:57:36.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj9 /dev/null 2024-09-16T16:57:36.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj10 /dev/null 2024-09-16T16:57:36.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj11 /dev/null 2024-09-16T16:57:36.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj12 /dev/null 2024-09-16T16:57:36.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj13 /dev/null 2024-09-16T16:57:36.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:36.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj14 /dev/null 2024-09-16T16:57:37.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:37.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj15 /dev/null 2024-09-16T16:57:37.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:37.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj16 /dev/null 2024-09-16T16:57:37.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:37.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj17 /dev/null 2024-09-16T16:57:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj18 /dev/null 2024-09-16T16:57:37.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:37.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj19 /dev/null 2024-09-16T16:57:37.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:37.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj20 /dev/null 2024-09-16T16:57:37.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:37.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj21 /dev/null 2024-09-16T16:57:37.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:37.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj22 /dev/null 2024-09-16T16:57:37.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:37.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj23 /dev/null 2024-09-16T16:57:37.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:37.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj24 /dev/null 2024-09-16T16:57:37.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:37.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj25 /dev/null 2024-09-16T16:57:38.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:38.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj26 /dev/null 2024-09-16T16:57:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:38.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj27 /dev/null 2024-09-16T16:57:38.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:38.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj28 /dev/null 2024-09-16T16:57:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:38.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj29 /dev/null 2024-09-16T16:57:38.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:38.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj30 /dev/null 2024-09-16T16:57:38.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:38.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj31 /dev/null 2024-09-16T16:57:38.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:38.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj32 /dev/null 2024-09-16T16:57:38.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:38.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj33 /dev/null 2024-09-16T16:57:38.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:38.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj34 /dev/null 2024-09-16T16:57:38.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:38.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj35 /dev/null 2024-09-16T16:57:38.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:38.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj36 /dev/null 2024-09-16T16:57:39.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:39.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj37 /dev/null 2024-09-16T16:57:39.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:39.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj38 /dev/null 2024-09-16T16:57:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:39.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj39 /dev/null 2024-09-16T16:57:39.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:39.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj40 /dev/null 2024-09-16T16:57:39.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:39.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj41 /dev/null 2024-09-16T16:57:39.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:39.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj42 /dev/null 2024-09-16T16:57:39.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:39.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj43 /dev/null 2024-09-16T16:57:39.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:39.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj44 /dev/null 2024-09-16T16:57:39.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:39.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj45 /dev/null 2024-09-16T16:57:39.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:39.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj46 /dev/null 2024-09-16T16:57:39.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:39.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj47 /dev/null 2024-09-16T16:57:40.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj48 /dev/null 2024-09-16T16:57:40.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj49 /dev/null 2024-09-16T16:57:40.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj50 /dev/null 2024-09-16T16:57:40.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj51 /dev/null 2024-09-16T16:57:40.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj52 /dev/null 2024-09-16T16:57:40.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj53 /dev/null 2024-09-16T16:57:40.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj54 /dev/null 2024-09-16T16:57:40.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj55 /dev/null 2024-09-16T16:57:40.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj56 /dev/null 2024-09-16T16:57:40.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj57 /dev/null 2024-09-16T16:57:40.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj58 /dev/null 2024-09-16T16:57:40.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj59 /dev/null 2024-09-16T16:57:40.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:40.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj60 /dev/null 2024-09-16T16:57:41.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:41.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj61 /dev/null 2024-09-16T16:57:41.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:41.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj62 /dev/null 2024-09-16T16:57:41.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:41.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj63 /dev/null 2024-09-16T16:57:41.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:41.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj64 /dev/null 2024-09-16T16:57:41.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:41.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj65 /dev/null 2024-09-16T16:57:41.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:41.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj66 /dev/null 2024-09-16T16:57:41.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:41.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj67 /dev/null 2024-09-16T16:57:41.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:41.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj68 /dev/null 2024-09-16T16:57:41.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:41.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj69 /dev/null 2024-09-16T16:57:41.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:41.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj70 /dev/null 2024-09-16T16:57:41.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:41.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj71 /dev/null 2024-09-16T16:57:42.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj72 /dev/null 2024-09-16T16:57:42.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj73 /dev/null 2024-09-16T16:57:42.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj74 /dev/null 2024-09-16T16:57:42.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj75 /dev/null 2024-09-16T16:57:42.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj76 /dev/null 2024-09-16T16:57:42.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj77 /dev/null 2024-09-16T16:57:42.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj78 /dev/null 2024-09-16T16:57:42.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj79 /dev/null 2024-09-16T16:57:42.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj80 /dev/null 2024-09-16T16:57:42.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj81 /dev/null 2024-09-16T16:57:42.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj82 /dev/null 2024-09-16T16:57:42.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:42.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj83 /dev/null 2024-09-16T16:57:43.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:43.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj84 /dev/null 2024-09-16T16:57:43.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:43.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj85 /dev/null 2024-09-16T16:57:43.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:43.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj86 /dev/null 2024-09-16T16:57:43.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:43.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj87 /dev/null 2024-09-16T16:57:43.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:43.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj88 /dev/null 2024-09-16T16:57:43.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:43.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj89 /dev/null 2024-09-16T16:57:43.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:43.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj90 /dev/null 2024-09-16T16:57:43.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:43.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj91 /dev/null 2024-09-16T16:57:43.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:43.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj92 /dev/null 2024-09-16T16:57:43.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:43.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj93 /dev/null 2024-09-16T16:57:43.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:43.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj94 /dev/null 2024-09-16T16:57:44.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj95 /dev/null 2024-09-16T16:57:44.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj96 /dev/null 2024-09-16T16:57:44.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj97 /dev/null 2024-09-16T16:57:44.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj98 /dev/null 2024-09-16T16:57:44.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj99 /dev/null 2024-09-16T16:57:44.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj100 /dev/null 2024-09-16T16:57:44.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj101 /dev/null 2024-09-16T16:57:44.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj102 /dev/null 2024-09-16T16:57:44.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj103 /dev/null 2024-09-16T16:57:44.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj104 /dev/null 2024-09-16T16:57:44.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj105 /dev/null 2024-09-16T16:57:44.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:44.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj106 /dev/null 2024-09-16T16:57:45.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:45.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj107 /dev/null 2024-09-16T16:57:45.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:45.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj108 /dev/null 2024-09-16T16:57:45.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:45.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj109 /dev/null 2024-09-16T16:57:45.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:45.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj110 /dev/null 2024-09-16T16:57:45.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:45.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj111 /dev/null 2024-09-16T16:57:45.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:45.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj112 /dev/null 2024-09-16T16:57:45.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:45.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj113 /dev/null 2024-09-16T16:57:45.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:45.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj114 /dev/null 2024-09-16T16:57:45.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:45.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj115 /dev/null 2024-09-16T16:57:45.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:45.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj116 /dev/null 2024-09-16T16:57:45.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:45.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj117 /dev/null 2024-09-16T16:57:46.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj118 /dev/null 2024-09-16T16:57:46.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj119 /dev/null 2024-09-16T16:57:46.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj120 /dev/null 2024-09-16T16:57:46.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj121 /dev/null 2024-09-16T16:57:46.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj122 /dev/null 2024-09-16T16:57:46.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj123 /dev/null 2024-09-16T16:57:46.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj124 /dev/null 2024-09-16T16:57:46.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj125 /dev/null 2024-09-16T16:57:46.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj126 /dev/null 2024-09-16T16:57:46.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj127 /dev/null 2024-09-16T16:57:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj128 /dev/null 2024-09-16T16:57:46.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj129 /dev/null 2024-09-16T16:57:46.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:46.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj130 /dev/null 2024-09-16T16:57:47.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:47.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj131 /dev/null 2024-09-16T16:57:47.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:47.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj132 /dev/null 2024-09-16T16:57:47.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:47.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj133 /dev/null 2024-09-16T16:57:47.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:47.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj134 /dev/null 2024-09-16T16:57:47.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:47.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj135 /dev/null 2024-09-16T16:57:47.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:47.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj136 /dev/null 2024-09-16T16:57:47.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:47.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj137 /dev/null 2024-09-16T16:57:47.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:47.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj138 /dev/null 2024-09-16T16:57:47.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:47.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj139 /dev/null 2024-09-16T16:57:47.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:47.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj140 /dev/null 2024-09-16T16:57:47.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:47.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj141 /dev/null 2024-09-16T16:57:48.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj142 /dev/null 2024-09-16T16:57:48.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj143 /dev/null 2024-09-16T16:57:48.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj144 /dev/null 2024-09-16T16:57:48.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj145 /dev/null 2024-09-16T16:57:48.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj146 /dev/null 2024-09-16T16:57:48.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj147 /dev/null 2024-09-16T16:57:48.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj148 /dev/null 2024-09-16T16:57:48.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj149 /dev/null 2024-09-16T16:57:48.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj150 /dev/null 2024-09-16T16:57:48.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj151 /dev/null 2024-09-16T16:57:48.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj152 /dev/null 2024-09-16T16:57:48.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:48.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj153 /dev/null 2024-09-16T16:57:49.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:49.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj154 /dev/null 2024-09-16T16:57:49.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:49.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj155 /dev/null 2024-09-16T16:57:49.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:49.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj156 /dev/null 2024-09-16T16:57:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:49.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj157 /dev/null 2024-09-16T16:57:49.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:49.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj158 /dev/null 2024-09-16T16:57:49.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:49.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj159 /dev/null 2024-09-16T16:57:49.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:49.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj160 /dev/null 2024-09-16T16:57:49.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:49.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj161 /dev/null 2024-09-16T16:57:49.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:49.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj162 /dev/null 2024-09-16T16:57:49.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:49.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj163 /dev/null 2024-09-16T16:57:50.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj164 /dev/null 2024-09-16T16:57:50.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj165 /dev/null 2024-09-16T16:57:50.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj166 /dev/null 2024-09-16T16:57:50.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj167 /dev/null 2024-09-16T16:57:50.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj168 /dev/null 2024-09-16T16:57:50.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj169 /dev/null 2024-09-16T16:57:50.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj170 /dev/null 2024-09-16T16:57:50.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj171 /dev/null 2024-09-16T16:57:50.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj172 /dev/null 2024-09-16T16:57:50.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj173 /dev/null 2024-09-16T16:57:50.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj174 /dev/null 2024-09-16T16:57:50.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:50.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj175 /dev/null 2024-09-16T16:57:51.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj176 /dev/null 2024-09-16T16:57:51.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj177 /dev/null 2024-09-16T16:57:51.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj178 /dev/null 2024-09-16T16:57:51.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj179 /dev/null 2024-09-16T16:57:51.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj180 /dev/null 2024-09-16T16:57:51.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj181 /dev/null 2024-09-16T16:57:51.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj182 /dev/null 2024-09-16T16:57:51.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj183 /dev/null 2024-09-16T16:57:51.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj184 /dev/null 2024-09-16T16:57:51.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj185 /dev/null 2024-09-16T16:57:51.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj186 /dev/null 2024-09-16T16:57:51.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj187 /dev/null 2024-09-16T16:57:51.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:51.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj188 /dev/null 2024-09-16T16:57:52.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:52.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj189 /dev/null 2024-09-16T16:57:52.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:52.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj190 /dev/null 2024-09-16T16:57:52.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:52.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj191 /dev/null 2024-09-16T16:57:52.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:52.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj192 /dev/null 2024-09-16T16:57:52.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:52.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj193 /dev/null 2024-09-16T16:57:52.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:52.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj194 /dev/null 2024-09-16T16:57:52.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:52.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj195 /dev/null 2024-09-16T16:57:52.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:52.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj196 /dev/null 2024-09-16T16:57:52.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:52.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj197 /dev/null 2024-09-16T16:57:52.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:52.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj198 /dev/null 2024-09-16T16:57:52.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:52.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj199 /dev/null 2024-09-16T16:57:53.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-16T16:57:53.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj200 /dev/null 2024-09-16T16:57:53.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: get_primary test obj1 2024-09-16T16:57:53.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:57:53.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:57:53.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:57:53.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:57:53.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: local primary=1 2024-09-16T16:57:53.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: get_pg test obj1 2024-09-16T16:57:53.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-16T16:57:53.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-16T16:57:53.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T16:57:53.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-16T16:57:54.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: local PG=1.0 2024-09-16T16:57:54.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:329: TEST_recovery_undersized: ceph osd set norecover 2024-09-16T16:57:54.854 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is set 2024-09-16T16:57:54.875 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: expr 3 - 1 2024-09-16T16:57:54.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: seq 0 2 2024-09-16T16:57:54.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-16T16:57:54.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:333: TEST_recovery_undersized: '[' 0 = 1 ']' 2024-09-16T16:57:54.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:337: TEST_recovery_undersized: ceph osd out osd.0 2024-09-16T16:57:55.616 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T16:57:55.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:338: TEST_recovery_undersized: break 2024-09-16T16:57:55.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:340: TEST_recovery_undersized: ceph osd pool set test size 4 2024-09-16T16:57:56.402 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 4 2024-09-16T16:57:56.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:341: TEST_recovery_undersized: ceph osd unset norecover 2024-09-16T16:57:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:norecover is unset 2024-09-16T16:57:57.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: get_primary test obj1 2024-09-16T16:57:57.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:57:57.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:57:57.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:57:57.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:57:58.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-16T16:57:58.338 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T16:57:58.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:344: TEST_recovery_undersized: sleep 10 2024-09-16T16:58:08.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:345: TEST_recovery_undersized: flush_pg_stats 2024-09-16T16:58:08.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T16:58:08.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T16:58:08.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T16:58:08.793 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T16:58:08.793 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T16:58:08.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T16:58:08.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:58:08.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T16:58:09.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836510 2024-09-16T16:58:09.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836510 2024-09-16T16:58:09.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510' 2024-09-16T16:58:09.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:58:09.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T16:58:09.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672984 2024-09-16T16:58:09.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672984 2024-09-16T16:58:09.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510 1-42949672984' 2024-09-16T16:58:09.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T16:58:09.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T16:58:09.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509457 2024-09-16T16:58:09.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509457 2024-09-16T16:58:09.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510 1-42949672984 2-64424509457' 2024-09-16T16:58:09.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:58:09.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836510 2024-09-16T16:58:09.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:58:09.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T16:58:09.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836510 2024-09-16T16:58:09.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:58:09.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836510 2024-09-16T16:58:09.466 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836510 2024-09-16T16:58:09.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-16T16:58:09.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T16:58:09.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-16T16:58:09.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:58:09.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672984 2024-09-16T16:58:09.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:58:09.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T16:58:09.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672984 2024-09-16T16:58:09.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:58:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672984 2024-09-16T16:58:09.928 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672984 2024-09-16T16:58:09.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-16T16:58:09.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:58:10.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672982 -lt 42949672984 2024-09-16T16:58:10.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T16:58:11.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T16:58:11.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T16:58:11.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672985 -lt 42949672984 2024-09-16T16:58:11.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T16:58:11.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509457 2024-09-16T16:58:11.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T16:58:11.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T16:58:11.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509457 2024-09-16T16:58:11.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T16:58:11.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509457 2024-09-16T16:58:11.789 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509457 2024-09-16T16:58:11.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-16T16:58:11.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T16:58:12.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-16T16:58:12.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: seq 1 300 2024-09-16T16:58:12.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 300) 2024-09-16T16:58:12.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2024-09-16T16:58:12.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2024-09-16T16:58:12.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2024-09-16T16:58:12.634 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T16:58:12.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:354: TEST_recovery_undersized: break 2024-09-16T16:58:12.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: get_primary test obj1 2024-09-16T16:58:12.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-16T16:58:12.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-16T16:58:12.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T16:58:12.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-16T16:58:13.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: primary=1 2024-09-16T16:58:13.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:366: TEST_recovery_undersized: local log=td/osd-recovery-stats/osd.1.log 2024-09-16T16:58:13.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: expr 200 '*' 3 2024-09-16T16:58:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: local first_degraded=600 2024-09-16T16:58:13.083 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: expr 200 '*' 2 2024-09-16T16:58:13.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: local last_degraded=400 2024-09-16T16:58:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:370: TEST_recovery_undersized: check td/osd-recovery-stats 1.0 1 replicated 600 400 0 0 2024-09-16T16:58:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-16T16:58:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-16T16:58:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-16T16:58:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-16T16:58:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=600 2024-09-16T16:58:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=400 2024-09-16T16:58:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-09-16T16:58:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-16T16:58:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-16T16:58:13.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-16T16:58:13.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-16T16:58:13.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-16T16:58:13.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-16T16:58:13.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-16T16:58:13.087 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-16T16:58:13.087 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:58:13.087 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-16T16:58:13.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2]' 2024-09-16T16:58:13.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-16T16:58:13.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:58:13.102 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:58:13.102 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2] ' 2024-09-16T16:58:13.103 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-16T16:58:13.103 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-16T16:58:13.103 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:58:13.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=600 2024-09-16T16:58:13.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 600 600 2024-09-16T16:58:13.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=600 2024-09-16T16:58:13.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:58:13.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=600 2024-09-16T16:58:13.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:58:13.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:58:13.109 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:58:13.109 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-16T16:58:13.110 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:58:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=400 2024-09-16T16:58:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 400 400 2024-09-16T16:58:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=400 2024-09-16T16:58:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:58:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=400 2024-09-16T16:58:13.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:58:13.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-16T16:58:13.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:58:13.120 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:58:13.121 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2] ' 2024-09-16T16:58:13.121 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-16T16:58:13.121 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-16T16:58:13.121 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:58:13.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-09-16T16:58:13.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-09-16T16:58:13.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-16T16:58:13.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-16T16:58:13.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-16T16:58:13.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-16T16:58:13.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-16T16:58:13.128 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-16T16:58:13.128 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-16T16:58:13.128 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-16T16:58:13.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-16T16:58:13.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-16T16:58:13.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-16T16:58:13.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-16T16:58:13.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-16T16:58:13.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-16T16:58:13.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-16T16:58:13.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:372: TEST_recovery_undersized: delete_pool test 2024-09-16T16:58:13.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T16:58:13.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T16:58:13.769 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T16:58:13.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:373: TEST_recovery_undersized: kill_daemons td/osd-recovery-stats 2024-09-16T16:58:13.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:58:13.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:58:13.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:58:13.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:58:13.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:58:19.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:58:19.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-16T16:58:19.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:58:19.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:58:19.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:58:19.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:58:19.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:58:19.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:58:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:58:19.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:58:19.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:58:19.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:58:19.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:58:19.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:58:19.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:58:19.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:58:19.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:58:19.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:58:19.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:58:19.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:58:19.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:58:19.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:58:19.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:58:19.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:58:19.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:58:19.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:58:19.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:58:19.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:58:19.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:58:19.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:58:19.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T16:58:19.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-recovery-stats 0 2024-09-16T16:58:19.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-16T16:58:19.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T16:58:19.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-16T16:58:19.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:58:19.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:58:19.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:58:19.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:58:19.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:58:19.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:58:19.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:58:19.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:58:19.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:58:19.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:58:19.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:58:19.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:58:19.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:58:19.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:58:19.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:58:19.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:58:19.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:58:19.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T16:58:19.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-16T16:58:19.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:58:19.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:58:19.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274941 2024-09-16T16:58:19.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274941 2024-09-16T16:58:19.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:58:19.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:58:19.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T16:58:19.161 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T16:58:19.161 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T16:58:19.171 INFO:tasks.workunit:Running workunit osd/osd-rep-recov-eio.sh... 2024-09-16T16:58:19.171 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-rep-recov-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh 2024-09-16T16:58:19.224 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T16:58:19.230 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T16:58:19.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T16:58:19.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T16:58:19.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T16:58:19.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T16:58:19.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T16:58:19.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T16:58:19.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T16:58:19.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T16:58:19.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-rep-recov-eio.sh 2024-09-16T16:58:19.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:25: run: local dir=td/osd-rep-recov-eio.sh 2024-09-16T16:58:19.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:26: run: shift 2024-09-16T16:58:19.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: export CEPH_MON=127.0.0.1:7140 2024-09-16T16:58:19.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: CEPH_MON=127.0.0.1:7140 2024-09-16T16:58:19.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:29: run: export CEPH_ARGS 2024-09-16T16:58:19.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: uuidgen 2024-09-16T16:58:19.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: CEPH_ARGS+='--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none ' 2024-09-16T16:58:19.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:31: run: CEPH_ARGS+='--mon-host=127.0.0.1:7140 ' 2024-09-16T16:58:19.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: set 2024-09-16T16:58:19.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T16:58:19.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: local 'funcs=TEST_rados_get_with_eio 2024-09-16T16:58:19.236 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_rados_repair_warning 2024-09-16T16:58:19.236 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_rep_backfill_unfound 2024-09-16T16:58:19.236 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_rep_recovery_unfound' 2024-09-16T16:58:19.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-16T16:58:19.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-16T16:58:19.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-16T16:58:19.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-16T16:58:19.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-16T16:58:19.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T16:58:19.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-16T16:58:19.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T16:58:19.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T16:58:19.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T16:58:19.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T16:58:19.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T16:58:19.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T16:58:19.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T16:58:19.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T16:58:19.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T16:58:19.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T16:58:19.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T16:58:19.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T16:58:19.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:58:19.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T16:58:19.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T16:58:19.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:58:19.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:58:19.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T16:58:19.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-16T16:58:19.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T16:58:19.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:58:19.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T16:58:19.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383662 2024-09-16T16:58:19.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T16:58:19.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T16:58:19.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-16T16:58:19.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T16:58:19.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:58:19.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T16:58:19.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.383662 2024-09-16T16:58:19.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T16:58:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T16:58:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T16:58:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T16:58:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-16T16:58:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-16T16:58:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-16T16:58:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T16:58:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T16:58:19.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T16:58:19.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-16T16:58:19.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-16T16:58:19.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T16:58:19.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:58:19.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:58:19.662 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:58:19.662 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:58:19.662 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T16:58:19.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T16:58:19.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-16T16:58:19.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T16:58:19.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T16:58:19.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:58:19.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:58:19.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T16:58:19.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T16:58:19.909 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:58:19.909 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:58:19.909 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:58:19.909 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:58:19.909 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:58:19.910 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T16:58:19.910 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-mon.a.asok 2024-09-16T16:58:19.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:58:19.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383662/ceph-mon.a.asok config get fsid 2024-09-16T16:58:20.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T16:58:20.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T16:58:20.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T16:58:20.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T16:58:20.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T16:58:20.031 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T16:58:20.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T16:58:20.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T16:58:20.032 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T16:58:20.032 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:58:20.032 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T16:58:20.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-mon.a.asok 2024-09-16T16:58:20.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T16:58:20.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383662/ceph-mon.a.asok config get mon_host 2024-09-16T16:58:20.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-16T16:58:20.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-16T16:58:20.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T16:58:20.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T16:58:20.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T16:58:20.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-16T16:58:20.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T16:58:21.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T16:58:21.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:58:21.071 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:58:21.072 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:58:21.072 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:58:21.072 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T16:58:21.072 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T16:58:21.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:58:21.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T16:58:21.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-16T16:58:21.443 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T16:58:21.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_get_with_eio td/osd-rep-recov-eio.sh 2024-09-16T16:58:21.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:168: TEST_rados_get_with_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T16:58:21.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:170: TEST_rados_get_with_eio: setup_osds 4 2024-09-16T16:58:21.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-09-16T16:58:21.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-16T16:58:21.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-16T16:58:21.461 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-09-16T16:58:21.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-09-16T16:58:21.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T16:58:21.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-16T16:58:21.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T16:58:21.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:58:21.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T16:58:21.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:58:21.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-16T16:58:21.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-16T16:58:21.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:58:21.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:58:21.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:58:21.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:58:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:58:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T16:58:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T16:58:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:58:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:58:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:58:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:58:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:58:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:58:21.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-16T16:58:21.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:58:21.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ac96af98-3977-4cc7-bba7-a0097a0bcf7f 2024-09-16T16:58:21.469 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 ac96af98-3977-4cc7-bba7-a0097a0bcf7f 2024-09-16T16:58:21.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ac96af98-3977-4cc7-bba7-a0097a0bcf7f' 2024-09-16T16:58:21.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:58:21.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtY+hmxUANHRAAT0pkpVQ8sR6DqHHAoaFpXg== 2024-09-16T16:58:21.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtY+hmxUANHRAAT0pkpVQ8sR6DqHHAoaFpXg=="}' 2024-09-16T16:58:21.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ac96af98-3977-4cc7-bba7-a0097a0bcf7f -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-16T16:58:21.840 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:58:21.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-16T16:58:21.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtY+hmxUANHRAAT0pkpVQ8sR6DqHHAoaFpXg== --osd-uuid ac96af98-3977-4cc7-bba7-a0097a0bcf7f 2024-09-16T16:58:21.890 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:21.885+0000 7f586205e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:58:21.892 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:21.889+0000 7f586205e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:58:21.893 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:21.889+0000 7f586205e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:58:21.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:21.889+0000 7f586205e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T16:58:21.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:21.889+0000 7f586205e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T16:58:21.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:21.889+0000 7f586205e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T16:58:21.971 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:21.969+0000 7f586205e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-16T16:58:27.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-16T16:58:27.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:58:27.882 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T16:58:27.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T16:58:27.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:58:28.431 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T16:58:28.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T16:58:28.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:58:28.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:58:28.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:58:28.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:58:28.463 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:28.461+0000 7f0aea2e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:58:28.465 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:28.461+0000 7f0aea2e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:58:28.466 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:28.461+0000 7f0aea2e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:58:28.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T16:58:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:58:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T16:58:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:58:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:58:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:28.922 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:58:28.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:58:28.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:28.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:29.024 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:29.021+0000 7f0aea2e57c0 -1 Falling back to public interface 2024-09-16T16:58:29.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:30.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:30.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:30.406 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:58:30.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:58:30.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:30.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:30.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:31.638 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:31.633+0000 7f0aea2e57c0 -1 osd.0 0 log_to_monitors true 2024-09-16T16:58:31.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:31.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:31.838 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:58:31.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:58:31.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:31.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:32.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:33.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:33.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:33.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:58:33.330 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:58:33.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:33.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:33.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:34.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:34.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:34.758 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:58:34.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:58:34.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:34.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:35.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:36.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:36.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:36.183 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:58:36.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:58:36.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:36.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:36.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:37.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:37.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:37.720 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:58:37.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:58:37.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:37.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:38.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:39.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:39.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:39.193 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:58:39.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:58:39.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:39.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:39.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:40.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:40.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:40.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:58:40.648 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:58:40.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:40.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:41.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:42.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:42.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:42.086 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:58:42.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:58:42.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:42.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:42.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:43.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:43.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:43.712 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:58:43.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:58:43.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:43.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:45.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:45.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:45.131 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:58:45.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:58:45.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:45.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:45.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:46.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:46.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:46.983 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:58:46.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:58:46.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:46.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:47.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:48.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:48.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:48.776 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:58:48.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:58:48.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:48.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:49.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:50.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:50.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:50.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:58:50.683 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:58:50.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:50.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:51.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:52.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:52.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:52.746 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:58:52.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:58:52.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:52.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:53.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:54.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:54.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:54.596 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:58:54.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:58:54.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:54.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:55.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:58:56.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:58:56.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:58:56.549 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:58:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:58:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:58:56.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T16:58:57.160 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/75325826,v1:127.0.0.1:6803/75325826] [v2:127.0.0.1:6804/75325826,v1:127.0.0.1:6805/75325826] exists,up ac96af98-3977-4cc7-bba7-a0097a0bcf7f 2024-09-16T16:58:57.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:58:57.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:58:57.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-16T16:58:57.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T16:58:57.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T16:58:57.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:58:57.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:58:57.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:58:57.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:58:57.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:58:57.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:58:57.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-16T16:58:57.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:58:57.167 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 9249db3a-ff13-4802-a94b-4380195eac20 2024-09-16T16:58:57.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9249db3a-ff13-4802-a94b-4380195eac20 2024-09-16T16:58:57.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9249db3a-ff13-4802-a94b-4380195eac20' 2024-09-16T16:58:57.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:58:57.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRY+hmUtvOChAAYHynLtp4upK/J4u+w5go4Q== 2024-09-16T16:58:57.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRY+hmUtvOChAAYHynLtp4upK/J4u+w5go4Q=="}' 2024-09-16T16:58:57.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9249db3a-ff13-4802-a94b-4380195eac20 -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-16T16:58:57.748 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:58:57.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-16T16:58:57.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRY+hmUtvOChAAYHynLtp4upK/J4u+w5go4Q== --osd-uuid 9249db3a-ff13-4802-a94b-4380195eac20 2024-09-16T16:58:57.808 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:57.804+0000 7f44f9e967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:58:57.829 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:57.824+0000 7f44f9e967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:58:57.830 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:57.824+0000 7f44f9e967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:58:57.831 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:57.828+0000 7f44f9e967c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T16:58:57.831 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:57.828+0000 7f44f9e967c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T16:58:57.831 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:57.828+0000 7f44f9e967c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T16:58:57.920 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:58:57.916+0000 7f44f9e967c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-16T16:59:04.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-16T16:59:04.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:59:04.157 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T16:59:04.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T16:59:04.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:59:04.718 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T16:59:04.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T16:59:04.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:59:04.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:59:04.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:59:04.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:59:04.748 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:04.744+0000 7f8cba81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:59:04.779 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:04.776+0000 7f8cba81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:59:04.780 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:04.776+0000 7f8cba81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:59:05.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T16:59:05.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:59:05.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T16:59:05.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:59:05.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:59:05.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:05.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:59:05.162 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:59:05.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:05.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:05.356 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:05.352+0000 7f8cba81b7c0 -1 Falling back to public interface 2024-09-16T16:59:05.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:06.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:06.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:06.603 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:59:06.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:59:06.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:06.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:07.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:07.903 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:07.900+0000 7f8cba81b7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T16:59:08.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:08.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:08.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:59:08.025 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:59:08.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:08.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:08.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:09.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:09.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:09.608 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:59:09.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:59:09.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:09.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:10.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:11.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:11.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:11.045 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:59:11.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:59:11.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:11.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:11.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:12.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:12.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:12.512 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:59:12.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:59:12.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:12.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:13.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:14.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:14.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:14.112 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:59:14.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:59:14.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:14.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:14.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:15.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:15.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:15.680 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:59:15.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:59:15.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:15.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:16.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:17.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:17.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:17.292 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:59:17.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:59:17.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:17.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:17.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:18.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:18.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:18.718 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:59:18.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:59:18.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:18.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:19.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:20.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:20.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:20.226 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:59:20.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:59:20.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:20.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:21.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:22.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:22.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:22.118 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:59:22.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:59:22.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:22.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:22.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:23.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:23.759 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T16:59:23.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:23.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T16:59:23.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:23.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:24.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:25.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:25.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:25.376 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T16:59:25.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T16:59:25.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:25.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:26.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:27.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:27.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:27.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T16:59:27.060 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T16:59:27.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:27.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:27.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:28.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:28.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:28.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T16:59:28.785 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T16:59:28.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:28.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:29.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:30.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:30.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:30.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T16:59:30.577 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T16:59:30.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:30.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:31.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:32.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:32.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:32.794 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T16:59:32.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T16:59:32.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:32.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:33.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:34.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:34.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:34.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T16:59:34.429 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T16:59:34.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:34.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T16:59:34.952 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1437558630,v1:127.0.0.1:6811/1437558630] [v2:127.0.0.1:6812/1437558630,v1:127.0.0.1:6813/1437558630] exists,up 9249db3a-ff13-4802-a94b-4380195eac20 2024-09-16T16:59:34.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T16:59:34.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T16:59:34.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-16T16:59:34.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T16:59:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T16:59:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T16:59:34.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T16:59:34.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T16:59:34.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T16:59:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T16:59:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T16:59:34.956 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T16:59:34.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T16:59:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-16T16:59:34.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T16:59:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9ca493da-48d8-4bf8-9217-8a926d133a52 2024-09-16T16:59:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9ca493da-48d8-4bf8-9217-8a926d133a52' 2024-09-16T16:59:34.959 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 9ca493da-48d8-4bf8-9217-8a926d133a52 2024-09-16T16:59:34.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T16:59:34.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2Y+hm9E1gOhAACouowBcI2a8ObpXgM94MXg== 2024-09-16T16:59:34.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2Y+hm9E1gOhAACouowBcI2a8ObpXgM94MXg=="}' 2024-09-16T16:59:34.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9ca493da-48d8-4bf8-9217-8a926d133a52 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-16T16:59:35.543 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:59:35.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-16T16:59:35.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2Y+hm9E1gOhAACouowBcI2a8ObpXgM94MXg== --osd-uuid 9ca493da-48d8-4bf8-9217-8a926d133a52 2024-09-16T16:59:35.598 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:35.592+0000 7f43899897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:59:35.600 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:35.596+0000 7f43899897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:59:35.601 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:35.596+0000 7f43899897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:59:35.602 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:35.596+0000 7f43899897c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T16:59:35.602 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:35.596+0000 7f43899897c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T16:59:35.602 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:35.596+0000 7f43899897c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T16:59:35.684 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:35.680+0000 7f43899897c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-16T16:59:42.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-16T16:59:42.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T16:59:42.062 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T16:59:42.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T16:59:42.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T16:59:42.662 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T16:59:42.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T16:59:42.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T16:59:42.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T16:59:42.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T16:59:42.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T16:59:42.693 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:42.688+0000 7f68516e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:59:42.705 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:42.700+0000 7f68516e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:59:42.707 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:42.704+0000 7f68516e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T16:59:43.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T16:59:43.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T16:59:43.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T16:59:43.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T16:59:43.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T16:59:43.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:43.091 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T16:59:43.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T16:59:43.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:43.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:59:43.280 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:43.276+0000 7f68516e57c0 -1 Falling back to public interface 2024-09-16T16:59:43.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:44.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:44.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:44.529 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T16:59:44.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T16:59:44.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:44.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:59:44.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:45.932 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T16:59:45.927+0000 7f68516e57c0 -1 osd.2 0 log_to_monitors true 2024-09-16T16:59:45.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:45.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:45.959 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T16:59:45.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T16:59:45.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:45.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:59:46.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:47.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:47.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:47.410 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T16:59:47.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T16:59:47.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:47.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:59:47.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:48.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:48.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:48.943 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T16:59:48.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T16:59:48.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:48.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:59:49.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:50.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:50.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:50.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T16:59:50.424 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T16:59:50.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:50.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:59:51.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:52.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:52.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:52.023 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T16:59:52.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T16:59:52.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:52.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:59:52.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:53.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:53.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:53.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T16:59:53.538 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T16:59:53.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:53.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:59:54.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:55.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:55.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:55.066 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T16:59:55.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T16:59:55.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:55.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:59:55.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:56.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:56.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:56.521 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T16:59:56.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T16:59:56.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:56.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:59:57.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:58.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:58.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:58.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T16:59:58.060 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T16:59:58.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:58.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T16:59:58.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T16:59:59.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T16:59:59.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T16:59:59.629 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T16:59:59.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T16:59:59.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T16:59:59.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:00:00.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:01.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:01.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:01.213 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:00:01.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:00:01.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:01.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:00:02.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:03.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:03.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:03.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:00:03.077 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:00:03.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:03.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:00:03.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:04.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:04.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:04.798 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:00:04.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:00:04.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:04.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:00:05.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:06.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:06.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:06.841 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:00:06.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:00:06.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:06.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:00:08.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:09.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:09.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:09.025 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:00:09.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:00:09.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:09.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:00:09.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:10.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:10.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:10.568 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:00:10.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:00:10.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:10.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:00:11.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:12.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:12.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:12.438 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:00:12.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T17:00:12.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:12.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:00:13.113 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3506879140,v1:127.0.0.1:6819/3506879140] [v2:127.0.0.1:6820/3506879140,v1:127.0.0.1:6821/3506879140] exists,up 9ca493da-48d8-4bf8-9217-8a926d133a52 2024-09-16T17:00:13.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:00:13.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:00:13.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:00:13.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T17:00:13.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-09-16T17:00:13.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:00:13.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:00:13.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T17:00:13.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:00:13.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-09-16T17:00:13.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:00:13.117 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:00:13.118 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:00:13.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:00:13.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:00:13.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:00:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:00:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:00:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:00:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:00:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:00:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:00:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:00:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:00:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:00:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:00:13.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-09-16T17:00:13.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:00:13.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ee558f23-218a-4788-ac38-ee3188b7e8d9 2024-09-16T17:00:13.121 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 ee558f23-218a-4788-ac38-ee3188b7e8d9 2024-09-16T17:00:13.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 ee558f23-218a-4788-ac38-ee3188b7e8d9' 2024-09-16T17:00:13.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:00:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdZOhmf36ECBAAIzUqbHGlKbNr7qUjq6hpZw== 2024-09-16T17:00:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdZOhmf36ECBAAIzUqbHGlKbNr7qUjq6hpZw=="}' 2024-09-16T17:00:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ee558f23-218a-4788-ac38-ee3188b7e8d9 -i td/osd-rep-recov-eio.sh/3/new.json 2024-09-16T17:00:13.824 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:00:13.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-09-16T17:00:13.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdZOhmf36ECBAAIzUqbHGlKbNr7qUjq6hpZw== --osd-uuid ee558f23-218a-4788-ac38-ee3188b7e8d9 2024-09-16T17:00:13.874 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:13.867+0000 7fd1cb1b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:00:13.877 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:13.871+0000 7fd1cb1b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:00:13.878 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:13.871+0000 7fd1cb1b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:00:13.878 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:13.871+0000 7fd1cb1b87c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-16T17:00:13.878 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:13.871+0000 7fd1cb1b87c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-16T17:00:13.878 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:13.871+0000 7fd1cb1b87c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-16T17:00:14.022 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:14.015+0000 7fd1cb1b87c0 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-09-16T17:00:20.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-09-16T17:00:20.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:00:20.418 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T17:00:20.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T17:00:20.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:00:20.989 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T17:00:20.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T17:00:20.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:00:20.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:00:20.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:00:20.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:00:21.021 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:21.015+0000 7f42444527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:00:21.054 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:21.047+0000 7f42444527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:00:21.056 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:21.051+0000 7f42444527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:00:21.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T17:00:21.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:00:21.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T17:00:21.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:00:21.434 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:00:21.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:00:21.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:21.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:00:21.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:21.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:21.620 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:21.615+0000 7f42444527c0 -1 Falling back to public interface 2024-09-16T17:00:21.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:22.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:22.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:22.880 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:00:22.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:00:22.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:22.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:23.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:24.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:24.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:24.296 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:00:24.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:00:24.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:24.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:24.356 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:00:24.355+0000 7f42444527c0 -1 osd.3 0 log_to_monitors true 2024-09-16T17:00:24.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:25.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:25.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:25.759 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:00:25.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:00:25.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:25.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:26.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:27.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:27.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:27.575 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:00:27.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:00:27.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:27.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:28.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:29.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:29.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:29.581 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:00:29.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:00:29.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:29.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:30.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:31.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:31.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:31.074 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:00:31.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:00:31.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:31.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:31.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:32.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:32.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:32.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:00:32.569 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:00:32.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:32.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:33.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:34.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:34.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:34.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:00:34.065 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:00:34.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:34.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:34.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:35.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:35.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:35.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:00:35.533 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:00:35.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:35.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:35.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:36.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:36.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:36.978 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:00:36.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:00:36.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:36.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:37.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:38.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:38.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:00:38.535 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:00:38.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:38.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:38.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:39.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:39.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:39.953 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:00:39.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:00:39.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:39.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:40.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:41.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:41.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:41.670 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:00:41.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:00:41.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:41.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:42.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:43.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:43.164 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:00:43.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:43.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:00:43.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:43.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:44.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:45.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:45.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:45.326 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:00:45.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:00:45.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:45.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:46.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:47.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:47.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:47.107 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:00:47.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:00:47.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:47.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:47.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:48.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:48.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:48.766 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:00:48.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:00:48.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:48.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:49.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:50.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:50.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:50.628 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:00:50.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T17:00:50.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:50.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:51.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:00:52.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:00:52.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:00:52.448 INFO:tasks.workunit.client.0.smithi039.stdout:19 2024-09-16T17:00:52.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T17:00:52.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:00:52.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:00:53.015 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1114445185,v1:127.0.0.1:6827/1114445185] [v2:127.0.0.1:6828/1114445185,v1:127.0.0.1:6829/1114445185] exists,up ee558f23-218a-4788-ac38-ee3188b7e8d9 2024-09-16T17:00:53.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:00:53.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:00:53.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:00:53.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-16T17:00:53.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:00:53.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:00:53.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:00:53.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:00:53.017 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:00:53.017 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:00:53.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:00:53.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:00:53.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:00:53.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:00:53.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:00:53.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:00:53.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:00:53.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:00:53.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:00:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:00:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:00:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:00:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:00:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:00:53.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:00:53.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:00:53.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803805 2024-09-16T17:00:53.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803805 2024-09-16T17:00:53.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803805' 2024-09-16T17:00:53.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:00:53.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:00:54.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607574 2024-09-16T17:00:54.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607574 2024-09-16T17:00:54.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803805 1-51539607574' 2024-09-16T17:00:54.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:00:54.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:00:54.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378638 2024-09-16T17:00:54.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378638 2024-09-16T17:00:54.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803805 1-51539607574 2-81604378638' 2024-09-16T17:00:54.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:00:54.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:00:54.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=137438953482 2024-09-16T17:00:54.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 137438953482 2024-09-16T17:00:54.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803805 1-51539607574 2-81604378638 3-137438953482' 2024-09-16T17:00:54.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:00:54.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803805 2024-09-16T17:00:54.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:00:54.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:00:54.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803805 2024-09-16T17:00:54.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:00:54.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803805 2024-09-16T17:00:54.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803805' 2024-09-16T17:00:54.415 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803805 2024-09-16T17:00:54.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:00:54.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803805 -lt 25769803805 2024-09-16T17:00:54.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:00:54.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607574 2024-09-16T17:00:54.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:00:54.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:00:54.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607574 2024-09-16T17:00:54.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:00:54.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607574 2024-09-16T17:00:54.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607574' 2024-09-16T17:00:54.992 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607574 2024-09-16T17:00:54.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:00:55.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607572 -lt 51539607574 2024-09-16T17:00:55.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:00:56.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:00:56.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:00:56.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607575 -lt 51539607574 2024-09-16T17:00:56.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:00:56.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378638 2024-09-16T17:00:56.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:00:56.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:00:56.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378638 2024-09-16T17:00:56.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:00:56.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378638 2024-09-16T17:00:56.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378638' 2024-09-16T17:00:56.917 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378638 2024-09-16T17:00:56.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:00:57.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378638 -lt 81604378638 2024-09-16T17:00:57.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:00:57.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-137438953482 2024-09-16T17:00:57.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:00:57.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:00:57.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-137438953482 2024-09-16T17:00:57.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:00:57.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=137438953482 2024-09-16T17:00:57.361 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 137438953482 2024-09-16T17:00:57.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 137438953482' 2024-09-16T17:00:57.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:00:57.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 137438953482 -lt 137438953482 2024-09-16T17:00:57.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:00:57.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:00:57.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:00:58.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T17:00:58.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:00:58.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:00:58.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:00:58.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:00:58.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:00:58.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:00:58.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:00:58.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:00:58.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:00:58.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:00:58.834 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:00:59.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T17:00:59.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:00:59.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:00:59.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:172: TEST_rados_get_with_eio: local poolname=pool-rep 2024-09-16T17:00:59.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:173: TEST_rados_get_with_eio: create_pool pool-rep 1 1 2024-09-16T17:00:59.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-09-16T17:01:00.020 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'pool-rep' already exists 2024-09-16T17:01:00.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T17:01:01.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:174: TEST_rados_get_with_eio: wait_for_clean 2024-09-16T17:01:01.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:01:01.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:01:01.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:01:01.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:01:01.041 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:01:01.041 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:01:01.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:01:01.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:01:01.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:01:01.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:01:01.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:01:01.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:01:01.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:01:01.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:01:01.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:01:01.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:01:01.631 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:01.631 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:01:01.631 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:01:01.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:01:01.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:01.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:01:01.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803808 2024-09-16T17:01:01.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803808 2024-09-16T17:01:01.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803808' 2024-09-16T17:01:01.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:01.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:01:01.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607578 2024-09-16T17:01:01.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607578 2024-09-16T17:01:01.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803808 1-51539607578' 2024-09-16T17:01:01.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:01.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:01:02.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378641 2024-09-16T17:01:02.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378641 2024-09-16T17:01:02.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803808 1-51539607578 2-81604378641' 2024-09-16T17:01:02.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:02.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:01:02.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=137438953486 2024-09-16T17:01:02.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 137438953486 2024-09-16T17:01:02.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803808 1-51539607578 2-81604378641 3-137438953486' 2024-09-16T17:01:02.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:02.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803808 2024-09-16T17:01:02.290 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:02.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:01:02.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803808 2024-09-16T17:01:02.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:02.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803808 2024-09-16T17:01:02.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-09-16T17:01:02.293 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803808 2024-09-16T17:01:02.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:02.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803808 -lt 25769803808 2024-09-16T17:01:02.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:02.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607578 2024-09-16T17:01:02.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:02.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:01:02.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607578 2024-09-16T17:01:02.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:02.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607578 2024-09-16T17:01:02.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607578' 2024-09-16T17:01:02.711 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607578 2024-09-16T17:01:02.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:01:03.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607576 -lt 51539607578 2024-09-16T17:01:03.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:01:04.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:01:04.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:01:04.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607578 -lt 51539607578 2024-09-16T17:01:04.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:04.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378641 2024-09-16T17:01:04.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:04.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:01:04.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378641 2024-09-16T17:01:04.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:04.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378641 2024-09-16T17:01:04.565 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378641 2024-09-16T17:01:04.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378641' 2024-09-16T17:01:04.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:01:05.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378642 -lt 81604378641 2024-09-16T17:01:05.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:05.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-137438953486 2024-09-16T17:01:05.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:05.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:01:05.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-137438953486 2024-09-16T17:01:05.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:05.011 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 137438953486 2024-09-16T17:01:05.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=137438953486 2024-09-16T17:01:05.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 137438953486' 2024-09-16T17:01:05.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:01:05.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 137438953486 -lt 137438953486 2024-09-16T17:01:05.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:01:05.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:05.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:01:06.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:01:06.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:01:06.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:01:06.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:01:06.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:01:06.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:01:06.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:01:06.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-16T17:01:06.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:01:06.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:06.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:06.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-16T17:01:06.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:01:06.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:01:06.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:175: TEST_rados_get_with_eio: rados_get_data eio td/osd-rep-recov-eio.sh 2024-09-16T17:01:06.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:111: rados_get_data: local inject=eio 2024-09-16T17:01:06.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:112: rados_get_data: shift 2024-09-16T17:01:06.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:113: rados_get_data: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:06.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:115: rados_get_data: local poolname=pool-rep 2024-09-16T17:01:06.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:116: rados_get_data: local objname=obj-eio-383662 2024-09-16T17:01:06.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: get_pg pool-rep obj-eio-383662 2024-09-16T17:01:06.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-09-16T17:01:06.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-eio-383662 2024-09-16T17:01:06.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-eio-383662 2024-09-16T17:01:06.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: local pgid=2.0 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:119: rados_get_data: rados_put td/osd-rep-recov-eio.sh pool-rep obj-eio-383662 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-eio-383662 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:01:07.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:01:07.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-eio-383662 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:01:07.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:120: rados_get_data: inject_eio rep data pool-rep obj-eio-383662 td/osd-rep-recov-eio.sh 0 2024-09-16T17:01:07.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:01:07.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:01:07.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:01:07.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:01:07.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:01:07.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:01:07.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-383662 2024-09-16T17:01:07.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:01:07.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:07.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:01:07.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:01:07.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:01:07.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-eio-383662 2024-09-16T17:01:07.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:01:07.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383662 2024-09-16T17:01:07.532 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383662 2024-09-16T17:01:07.532 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:01:07.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:01:07.970 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:07.970 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:01:07.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:01:07.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:01:07.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:01:07.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:01:07.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:01:07.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:01:07.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:01:07.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:01:07.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:01:07.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:01:07.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:01:07.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:01:07.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:01:07.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:01:07.973 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:01:07.973 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:01:07.973 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:01:07.974 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:07.974 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:07.974 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:07.974 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:01:07.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:01:08.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:01:08.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:01:08.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:01:08.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:01:08.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:01:08.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:01:08.087 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:08.087 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:08.087 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:08.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:01:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:01:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-eio-383662 2024-09-16T17:01:08.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:121: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-383662 2024-09-16T17:01:08.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:08.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:01:08.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-383662 2024-09-16T17:01:08.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:01:08.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:01:08.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:01:08.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-383662 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:01:08.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:01:08.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:01:08.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:123: rados_get_data: wait_for_clean 2024-09-16T17:01:08.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:01:08.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:01:08.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:01:08.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:01:08.298 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:01:08.298 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:01:08.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:01:08.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:01:08.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:01:08.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:01:08.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:01:08.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:01:08.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:01:08.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:01:08.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:01:08.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:01:08.831 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:08.832 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:01:08.832 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:01:08.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:01:08.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:08.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:01:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803812 2024-09-16T17:01:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803812 2024-09-16T17:01:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803812' 2024-09-16T17:01:09.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:09.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:01:09.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607581 2024-09-16T17:01:09.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607581 2024-09-16T17:01:09.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803812 1-51539607581' 2024-09-16T17:01:09.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:09.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:01:09.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378645 2024-09-16T17:01:09.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378645 2024-09-16T17:01:09.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803812 1-51539607581 2-81604378645' 2024-09-16T17:01:09.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:09.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:01:09.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=137438953489 2024-09-16T17:01:09.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 137438953489 2024-09-16T17:01:09.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803812 1-51539607581 2-81604378645 3-137438953489' 2024-09-16T17:01:09.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:09.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:09.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803812 2024-09-16T17:01:09.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:01:09.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803812 2024-09-16T17:01:09.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:09.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803812 2024-09-16T17:01:09.476 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803812 2024-09-16T17:01:09.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-16T17:01:09.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:09.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803810 -lt 25769803812 2024-09-16T17:01:09.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:01:10.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:01:10.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:11.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-16T17:01:11.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:11.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607581 2024-09-16T17:01:11.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:11.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:01:11.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607581 2024-09-16T17:01:11.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:11.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607581 2024-09-16T17:01:11.319 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607581 2024-09-16T17:01:11.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607581' 2024-09-16T17:01:11.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:01:11.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607582 -lt 51539607581 2024-09-16T17:01:11.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:11.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:11.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378645 2024-09-16T17:01:11.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:01:11.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378645 2024-09-16T17:01:11.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:11.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378645 2024-09-16T17:01:11.756 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378645 2024-09-16T17:01:11.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378645' 2024-09-16T17:01:11.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:01:12.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378645 -lt 81604378645 2024-09-16T17:01:12.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:12.206 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-137438953489 2024-09-16T17:01:12.206 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:12.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:01:12.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-137438953489 2024-09-16T17:01:12.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:12.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=137438953489 2024-09-16T17:01:12.210 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 137438953489 2024-09-16T17:01:12.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 137438953489' 2024-09-16T17:01:12.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:01:12.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 137438953490 -lt 137438953489 2024-09-16T17:01:12.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:01:12.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:12.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:13.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:01:13.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:01:13.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:01:13.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:01:13.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:01:13.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:01:13.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:01:13.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:01:13.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-16T17:01:13.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:01:13.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:13.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:14.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-16T17:01:14.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:01:14.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:01:14.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-09-16T17:01:14.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: ceph pg 2.0 query 2024-09-16T17:01:14.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: COUNT=1 2024-09-16T17:01:14.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:125: rados_get_data: test 1 = 1 2024-09-16T17:01:14.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:126: rados_get_data: flush_pg_stats 2024-09-16T17:01:14.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:01:14.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:01:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:01:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:01:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:01:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:01:14.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:14.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:01:14.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803815 2024-09-16T17:01:14.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803815 2024-09-16T17:01:14.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803815' 2024-09-16T17:01:14.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:14.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:01:15.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607585 2024-09-16T17:01:15.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607585 2024-09-16T17:01:15.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803815 1-51539607585' 2024-09-16T17:01:15.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:15.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:01:15.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378648 2024-09-16T17:01:15.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378648 2024-09-16T17:01:15.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803815 1-51539607585 2-81604378648' 2024-09-16T17:01:15.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:15.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:01:15.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=137438953492 2024-09-16T17:01:15.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 137438953492 2024-09-16T17:01:15.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803815 1-51539607585 2-81604378648 3-137438953492' 2024-09-16T17:01:15.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:15.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803815 2024-09-16T17:01:15.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:15.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:01:15.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803815 2024-09-16T17:01:15.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:15.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803815 2024-09-16T17:01:15.422 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803815 2024-09-16T17:01:15.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2024-09-16T17:01:15.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:15.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803813 -lt 25769803815 2024-09-16T17:01:15.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:01:16.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:01:16.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803815 -lt 25769803815 2024-09-16T17:01:17.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:17.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607585 2024-09-16T17:01:17.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:17.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:01:17.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607585 2024-09-16T17:01:17.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:17.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607585 2024-09-16T17:01:17.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607585' 2024-09-16T17:01:17.243 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607585 2024-09-16T17:01:17.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:01:17.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607585 -lt 51539607585 2024-09-16T17:01:17.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:17.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378648 2024-09-16T17:01:17.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:17.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:01:17.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378648 2024-09-16T17:01:17.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:17.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378648 2024-09-16T17:01:17.682 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378648 2024-09-16T17:01:17.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378648' 2024-09-16T17:01:17.682 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:01:18.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378649 -lt 81604378648 2024-09-16T17:01:18.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:18.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-137438953492 2024-09-16T17:01:18.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:18.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:01:18.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:18.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-137438953492 2024-09-16T17:01:18.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=137438953492 2024-09-16T17:01:18.119 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 137438953492 2024-09-16T17:01:18.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 137438953492' 2024-09-16T17:01:18.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:01:18.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 137438953493 -lt 137438953492 2024-09-16T17:01:18.545 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: ceph pg dump --format=json-pretty 2024-09-16T17:01:18.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-16T17:01:18.965 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-09-16T17:01:18.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: COUNT=1 2024-09-16T17:01:18.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:128: rados_get_data: test 1 = 1 2024-09-16T17:01:18.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: get_osds pool-rep obj-eio-383662 2024-09-16T17:01:18.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:01:18.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383662 2024-09-16T17:01:18.984 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383662 2024-09-16T17:01:18.984 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:01:19.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:01:19.425 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:19.425 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:01:19.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:01:19.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: object_osds=('3' '1' '0') 2024-09-16T17:01:19.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: local object_osds 2024-09-16T17:01:19.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:131: rados_get_data: local primary=3 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:132: rados_get_data: local bad_peer=1 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:133: rados_get_data: inject_eio rep data pool-rep obj-eio-383662 td/osd-rep-recov-eio.sh 0 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-383662 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:01:19.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:01:19.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-eio-383662 2024-09-16T17:01:19.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:01:19.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383662 2024-09-16T17:01:19.427 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383662 2024-09-16T17:01:19.427 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:01:19.882 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:01:19.883 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:19.883 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:01:19.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:01:19.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:01:19.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:01:19.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:01:19.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:01:19.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:01:19.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:01:19.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:01:19.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:01:19.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:01:19.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:01:19.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:01:19.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:01:19.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:01:19.886 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:01:19.886 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:01:19.886 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:01:19.886 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:19.886 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:19.886 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:19.886 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:01:19.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:01:20.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:01:20.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:01:20.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:01:20.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:01:20.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:01:20.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:01:20.014 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:20.014 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:20.014 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:20.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:01:20.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:01:20.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-eio-383662 2024-09-16T17:01:20.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:134: rados_get_data: inject_eio rep data pool-rep obj-eio-383662 td/osd-rep-recov-eio.sh 1 2024-09-16T17:01:20.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:01:20.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:01:20.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:01:20.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:01:20.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:01:20.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:01:20.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-383662 2024-09-16T17:01:20.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:01:20.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:20.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:01:20.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:01:20.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:01:20.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-eio-383662 2024-09-16T17:01:20.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:01:20.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383662 2024-09-16T17:01:20.137 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383662 2024-09-16T17:01:20.137 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:01:20.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:01:20.612 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:20.612 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:01:20.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:01:20.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:01:20.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:01:20.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:01:20.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:01:20.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:01:20.613 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:01:20.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:01:20.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:01:20.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:01:20.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:01:20.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:01:20.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:01:20.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:01:20.616 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:01:20.616 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:01:20.616 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:01:20.616 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:20.616 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:20.616 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:20.617 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:01:20.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:01:20.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:01:20.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:01:20.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:01:20.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:01:20.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:01:20.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:01:20.738 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:20.738 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:20.738 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:20.739 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:01:20.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:01:20.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-eio-383662 2024-09-16T17:01:20.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:136: rados_get_data: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:01:20.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:01:20.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:01:20.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:01:20.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:01:20.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:01:20.867 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:01:20.867 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:01:20.867 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:01:20.868 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:20.868 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:20.868 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:20.868 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:01:20.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:01:20.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:01:20.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:137: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-383662 2024-09-16T17:01:20.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:20.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:01:20.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-383662 2024-09-16T17:01:20.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:01:20.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:01:20.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:01:20.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-383662 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:01:21.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:01:21.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:01:21.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:140: rados_get_data: wait_for_clean 2024-09-16T17:01:21.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:01:21.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:01:21.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:01:21.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:01:21.027 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:01:21.027 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:01:21.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:01:21.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:01:21.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:01:21.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:01:21.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:01:21.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:01:21.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:01:21.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:01:21.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:01:21.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:01:21.546 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:21.546 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:01:21.546 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:01:21.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:01:21.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:21.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:01:21.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803818 2024-09-16T17:01:21.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803818 2024-09-16T17:01:21.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803818' 2024-09-16T17:01:21.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:21.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:01:21.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607588 2024-09-16T17:01:21.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607588 2024-09-16T17:01:21.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803818 1-51539607588' 2024-09-16T17:01:21.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:21.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:01:21.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378651 2024-09-16T17:01:21.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378651 2024-09-16T17:01:21.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803818 1-51539607588 2-81604378651' 2024-09-16T17:01:21.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:21.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:01:22.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=137438953496 2024-09-16T17:01:22.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 137438953496 2024-09-16T17:01:22.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803818 1-51539607588 2-81604378651 3-137438953496' 2024-09-16T17:01:22.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:22.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803818 2024-09-16T17:01:22.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:22.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:01:22.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803818 2024-09-16T17:01:22.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:22.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803818 2024-09-16T17:01:22.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803818' 2024-09-16T17:01:22.156 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803818 2024-09-16T17:01:22.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:22.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803818 -lt 25769803818 2024-09-16T17:01:22.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:22.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607588 2024-09-16T17:01:22.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:22.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:01:22.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607588 2024-09-16T17:01:22.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:22.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607588 2024-09-16T17:01:22.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607588' 2024-09-16T17:01:22.562 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607588 2024-09-16T17:01:22.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:01:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607588 -lt 51539607588 2024-09-16T17:01:22.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:22.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:22.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378651 2024-09-16T17:01:22.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:01:22.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378651 2024-09-16T17:01:22.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:22.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378651 2024-09-16T17:01:22.980 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378651 2024-09-16T17:01:22.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378651' 2024-09-16T17:01:22.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:01:23.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378649 -lt 81604378651 2024-09-16T17:01:23.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:01:24.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:01:24.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:01:24.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378652 -lt 81604378651 2024-09-16T17:01:24.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:24.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-137438953496 2024-09-16T17:01:24.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:24.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:01:24.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-137438953496 2024-09-16T17:01:24.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:24.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=137438953496 2024-09-16T17:01:24.854 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 137438953496 2024-09-16T17:01:24.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 137438953496' 2024-09-16T17:01:24.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:01:25.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 137438953496 -lt 137438953496 2024-09-16T17:01:25.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:01:25.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:25.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:25.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:01:25.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:01:25.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:01:25.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:01:25.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:01:25.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:01:25.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:01:25.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:01:26.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-16T17:01:26.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:01:26.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:26.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:26.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-16T17:01:26.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:01:26.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:01:26.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:142: rados_get_data: inject_eio rep data pool-rep obj-eio-383662 td/osd-rep-recov-eio.sh 0 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-383662 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:01:26.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:01:26.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-eio-383662 2024-09-16T17:01:26.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:01:26.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383662 2024-09-16T17:01:26.795 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383662 2024-09-16T17:01:26.795 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:01:27.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:01:27.211 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:27.211 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:01:27.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:01:27.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:01:27.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:01:27.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:01:27.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:01:27.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:01:27.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:01:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:01:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:01:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:01:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:01:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:01:27.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:01:27.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:01:27.214 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:01:27.214 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:01:27.214 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:01:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:27.215 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:01:27.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:01:27.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:01:27.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:01:27.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:01:27.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:01:27.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:01:27.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:01:27.337 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:27.337 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:27.337 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:27.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:01:27.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:01:27.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-eio-383662 2024-09-16T17:01:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:143: rados_get_data: inject_eio rep data pool-rep obj-eio-383662 td/osd-rep-recov-eio.sh 2 2024-09-16T17:01:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:01:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:01:27.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-383662 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=2 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-eio-383662 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:01:27.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383662 2024-09-16T17:01:27.452 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383662 2024-09-16T17:01:27.452 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:01:27.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:01:27.858 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:27.858 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=0 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:01:27.859 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-16T17:01:27.860 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T17:01:27.860 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T17:01:27.860 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:27.860 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:27.860 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:27.860 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.0.asok 2024-09-16T17:01:27.860 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:01:27.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:01:27.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:01:27.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:01:27.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.0 2024-09-16T17:01:27.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T17:01:27.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T17:01:27.982 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:27.982 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:27.982 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:27.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.0.asok 2024-09-16T17:01:27.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:01:27.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.0.asok injectdataerr pool-rep obj-eio-383662 2024-09-16T17:01:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:144: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-383662 2024-09-16T17:01:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:01:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-383662 2024-09-16T17:01:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:01:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:01:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:01:28.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-383662 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:01:28.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:01:28.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:01:28.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:146: rados_get_data: wait_for_clean 2024-09-16T17:01:28.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:01:28.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:01:28.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:01:28.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:01:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:01:28.153 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:01:28.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:01:28.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:01:28.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:01:28.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:01:28.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:01:28.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:01:28.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:01:28.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:01:28.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:01:28.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:01:28.746 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:28.746 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:01:28.746 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:01:28.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:01:28.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:28.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:01:28.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803822 2024-09-16T17:01:28.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803822 2024-09-16T17:01:28.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822' 2024-09-16T17:01:28.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:28.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:01:29.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607591 2024-09-16T17:01:29.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607591 2024-09-16T17:01:29.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822 1-51539607591' 2024-09-16T17:01:29.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:29.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:01:29.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378655 2024-09-16T17:01:29.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378655 2024-09-16T17:01:29.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822 1-51539607591 2-81604378655' 2024-09-16T17:01:29.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:29.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:01:29.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=137438953499 2024-09-16T17:01:29.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 137438953499 2024-09-16T17:01:29.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822 1-51539607591 2-81604378655 3-137438953499' 2024-09-16T17:01:29.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:29.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803822 2024-09-16T17:01:29.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:29.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:01:29.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803822 2024-09-16T17:01:29.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:29.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803822 2024-09-16T17:01:29.400 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803822 2024-09-16T17:01:29.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803822' 2024-09-16T17:01:29.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:29.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803820 -lt 25769803822 2024-09-16T17:01:29.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:01:30.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:01:30.817 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:31.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803822 -lt 25769803822 2024-09-16T17:01:31.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:31.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607591 2024-09-16T17:01:31.241 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:31.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:01:31.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:31.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607591 2024-09-16T17:01:31.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607591 2024-09-16T17:01:31.244 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607591 2024-09-16T17:01:31.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607591' 2024-09-16T17:01:31.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:01:31.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607592 -lt 51539607591 2024-09-16T17:01:31.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:31.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378655 2024-09-16T17:01:31.669 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:31.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:01:31.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378655 2024-09-16T17:01:31.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:31.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378655 2024-09-16T17:01:31.674 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378655 2024-09-16T17:01:31.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378655' 2024-09-16T17:01:31.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:01:32.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378655 -lt 81604378655 2024-09-16T17:01:32.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:32.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-137438953499 2024-09-16T17:01:32.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:32.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:01:32.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-137438953499 2024-09-16T17:01:32.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:32.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=137438953499 2024-09-16T17:01:32.102 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 137438953499 2024-09-16T17:01:32.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 137438953499' 2024-09-16T17:01:32.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:01:32.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 137438953500 -lt 137438953499 2024-09-16T17:01:32.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:01:32.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:32.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:33.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:01:33.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:01:33.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:01:33.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:01:33.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:01:33.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:01:33.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:01:33.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:01:33.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-16T17:01:33.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:01:33.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:33.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:34.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-16T17:01:34.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:01:34.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:01:34.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: ceph pg 2.0 query 2024-09-16T17:01:34.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-09-16T17:01:34.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: COUNT=3 2024-09-16T17:01:34.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:148: rados_get_data: test 3 = 3 2024-09-16T17:01:34.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:149: rados_get_data: flush_pg_stats 2024-09-16T17:01:34.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:01:34.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:01:34.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:01:34.600 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:34.601 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:01:34.601 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:01:34.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:01:34.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:34.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:01:34.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803825 2024-09-16T17:01:34.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803825 2024-09-16T17:01:34.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803825' 2024-09-16T17:01:34.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:34.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:01:34.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607595 2024-09-16T17:01:34.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607595 2024-09-16T17:01:34.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803825 1-51539607595' 2024-09-16T17:01:34.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:34.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:01:35.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378658 2024-09-16T17:01:35.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378658 2024-09-16T17:01:35.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803825 1-51539607595 2-81604378658' 2024-09-16T17:01:35.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:35.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:01:35.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=137438953502 2024-09-16T17:01:35.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 137438953502 2024-09-16T17:01:35.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803825 1-51539607595 2-81604378658 3-137438953502' 2024-09-16T17:01:35.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:35.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803825 2024-09-16T17:01:35.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:35.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:01:35.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803825 2024-09-16T17:01:35.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:35.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803825 2024-09-16T17:01:35.216 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803825 2024-09-16T17:01:35.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-09-16T17:01:35.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:35.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803823 -lt 25769803825 2024-09-16T17:01:35.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:01:36.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:01:36.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:37.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803825 -lt 25769803825 2024-09-16T17:01:37.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:37.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607595 2024-09-16T17:01:37.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:37.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:01:37.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607595 2024-09-16T17:01:37.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:37.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607595 2024-09-16T17:01:37.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607595' 2024-09-16T17:01:37.040 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607595 2024-09-16T17:01:37.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:01:37.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607595 -lt 51539607595 2024-09-16T17:01:37.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:37.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378658 2024-09-16T17:01:37.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:37.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:01:37.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378658 2024-09-16T17:01:37.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:37.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378658 2024-09-16T17:01:37.465 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378658 2024-09-16T17:01:37.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378658' 2024-09-16T17:01:37.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:01:37.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378658 -lt 81604378658 2024-09-16T17:01:37.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:37.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-137438953502 2024-09-16T17:01:37.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:37.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:01:37.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-137438953502 2024-09-16T17:01:37.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:37.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=137438953502 2024-09-16T17:01:37.925 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 137438953502 2024-09-16T17:01:37.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 137438953502' 2024-09-16T17:01:37.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:01:38.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 137438953503 -lt 137438953502 2024-09-16T17:01:38.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: ceph pg dump --format=json-pretty 2024-09-16T17:01:38.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-16T17:01:38.783 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-09-16T17:01:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: COUNT=4 2024-09-16T17:01:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:151: rados_get_data: test 4 = 4 2024-09-16T17:01:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:153: rados_get_data: inject_eio rep data pool-rep obj-eio-383662 td/osd-rep-recov-eio.sh 0 2024-09-16T17:01:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:01:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:01:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:01:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:01:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:01:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:01:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-383662 2024-09-16T17:01:38.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:01:38.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:38.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:01:38.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:01:38.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:01:38.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-eio-383662 2024-09-16T17:01:38.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:01:38.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383662 2024-09-16T17:01:38.803 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383662 2024-09-16T17:01:38.803 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:01:39.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:01:39.219 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:39.219 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:01:39.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:01:39.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:01:39.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:01:39.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:01:39.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:01:39.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:01:39.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:01:39.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:01:39.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:01:39.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:01:39.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:01:39.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:01:39.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:01:39.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:01:39.223 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:01:39.223 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:01:39.223 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:01:39.223 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:39.223 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:39.223 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:39.224 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:01:39.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:01:39.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:01:39.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:01:39.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:01:39.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:01:39.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:01:39.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:01:39.349 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:39.349 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:39.349 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:39.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:01:39.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:01:39.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-eio-383662 2024-09-16T17:01:39.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:154: rados_get_data: inject_eio rep data pool-rep obj-eio-383662 td/osd-rep-recov-eio.sh 1 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-383662 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:01:39.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:01:39.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-eio-383662 2024-09-16T17:01:39.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:01:39.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383662 2024-09-16T17:01:39.471 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:01:39.471 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383662 2024-09-16T17:01:39.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:01:39.895 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:39.895 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:01:39.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:01:39.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:01:39.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:01:39.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:01:39.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:01:39.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:01:39.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:01:39.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:01:39.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:01:39.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:01:39.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:01:39.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:01:39.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:01:39.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:01:39.898 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:01:39.898 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:01:39.898 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:01:39.899 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:39.899 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:39.899 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:39.899 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:01:39.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:01:40.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:01:40.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:01:40.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:01:40.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:01:40.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:01:40.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:01:40.029 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:40.029 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:40.029 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:40.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:01:40.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:01:40.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-eio-383662 2024-09-16T17:01:40.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:155: rados_get_data: inject_eio rep data pool-rep obj-eio-383662 td/osd-rep-recov-eio.sh 2 2024-09-16T17:01:40.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:01:40.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:01:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:01:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:01:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:01:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:01:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-383662 2024-09-16T17:01:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:01:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:01:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=2 2024-09-16T17:01:40.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:01:40.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-eio-383662 2024-09-16T17:01:40.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:01:40.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383662 2024-09-16T17:01:40.151 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383662 2024-09-16T17:01:40.151 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:01:40.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:01:40.582 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:40.582 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:01:40.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:01:40.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:01:40.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:01:40.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=0 2024-09-16T17:01:40.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:01:40.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:01:40.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-09-16T17:01:40.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:01:40.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-16T17:01:40.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:01:40.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-16T17:01:40.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:01:40.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:01:40.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:01:40.585 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-16T17:01:40.585 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T17:01:40.586 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T17:01:40.586 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:40.586 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:40.586 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:40.586 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.0.asok 2024-09-16T17:01:40.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:01:40.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:01:40.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:01:40.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:01:40.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.0 2024-09-16T17:01:40.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T17:01:40.706 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T17:01:40.706 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:01:40.706 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:01:40.707 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:01:40.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.0.asok 2024-09-16T17:01:40.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:01:40.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.0.asok injectdataerr pool-rep obj-eio-383662 2024-09-16T17:01:40.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:156: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-383662 hang 2024-09-16T17:01:40.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:01:40.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:01:40.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-383662 2024-09-16T17:01:40.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=hang 2024-09-16T17:01:40.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' hang = fail ']' 2024-09-16T17:01:40.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' hang = hang ']' 2024-09-16T17:01:40.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:98: rados_get: timeout 5 rados --pool pool-rep get obj-eio-383662 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:01:45.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:99: rados_get: test 124 = 124 2024-09-16T17:01:45.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:100: rados_get: return 2024-09-16T17:01:45.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:158: rados_get_data: wait_for_clean 2024-09-16T17:01:45.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:01:45.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:01:45.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:01:45.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:01:45.829 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:01:45.830 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:01:45.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:01:45.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:01:45.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:01:45.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:01:45.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:01:45.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:01:45.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:01:45.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:01:45.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:01:46.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:01:46.395 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:01:46.395 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:01:46.395 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:01:46.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:01:46.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:46.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:01:46.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803829 2024-09-16T17:01:46.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803829 2024-09-16T17:01:46.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803829' 2024-09-16T17:01:46.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:46.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:01:46.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607599 2024-09-16T17:01:46.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607599 2024-09-16T17:01:46.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803829 1-51539607599' 2024-09-16T17:01:46.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:46.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:01:46.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378662 2024-09-16T17:01:46.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378662 2024-09-16T17:01:46.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803829 1-51539607599 2-81604378662' 2024-09-16T17:01:46.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:01:46.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:01:47.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=137438953507 2024-09-16T17:01:47.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 137438953507 2024-09-16T17:01:47.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803829 1-51539607599 2-81604378662 3-137438953507' 2024-09-16T17:01:47.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:47.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803829 2024-09-16T17:01:47.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:47.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:01:47.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803829 2024-09-16T17:01:47.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:47.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803829 2024-09-16T17:01:47.029 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803829 2024-09-16T17:01:47.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-09-16T17:01:47.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:47.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803827 -lt 25769803829 2024-09-16T17:01:47.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:01:48.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:01:48.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:01:48.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803830 -lt 25769803829 2024-09-16T17:01:48.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:48.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607599 2024-09-16T17:01:48.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:48.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:01:48.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607599 2024-09-16T17:01:48.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:48.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607599 2024-09-16T17:01:48.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607599' 2024-09-16T17:01:48.909 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607599 2024-09-16T17:01:48.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:01:49.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607599 -lt 51539607599 2024-09-16T17:01:49.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:49.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378662 2024-09-16T17:01:49.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:01:49.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378662 2024-09-16T17:01:49.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:49.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378662 2024-09-16T17:01:49.348 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378662 2024-09-16T17:01:49.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378662' 2024-09-16T17:01:49.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:01:49.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378663 -lt 81604378662 2024-09-16T17:01:49.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:01:49.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-137438953507 2024-09-16T17:01:49.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:01:49.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:01:49.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:01:49.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-137438953507 2024-09-16T17:01:49.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=137438953507 2024-09-16T17:01:49.769 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 137438953507 2024-09-16T17:01:49.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 137438953507' 2024-09-16T17:01:49.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:01:50.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 137438953507 -lt 137438953507 2024-09-16T17:01:50.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:01:50.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:50.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:50.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:01:50.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:01:50.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:01:50.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:01:50.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:01:50.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:01:50.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:01:50.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:01:51.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:01:51.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:01:51.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:51.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:51.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:01:51.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-09-16T17:01:51.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T17:01:51.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-09-16T17:01:51.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:01:51.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T17:01:51.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:01:51.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:01:51.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:01:51.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:01:51.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:01:51.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:01:51.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:01:51.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:01:52.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:01:52.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:01:52.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:52.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:52.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:01:52.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:01:52.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:01:52.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:01:52.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:01:52.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:01:52.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:01:52.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:01:52.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:01:53.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:01:53.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:01:53.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-16T17:01:53.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:01:53.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T17:01:53.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:01:53.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:01:53.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:01:53.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:01:53.522 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:01:53.522 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:01:53.522 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:01:53.522 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:01:53.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:01:53.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:01:53.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:53.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:54.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:01:54.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:01:54.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:01:54.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:01:54.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:01:54.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:01:54.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:01:54.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:01:54.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:01:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:01:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:01:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-09-16T17:01:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:01:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T17:01:55.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:01:55.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:01:55.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:01:55.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:01:55.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:01:55.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:01:55.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:01:55.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:01:55.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:01:55.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:01:55.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:55.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:56.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:01:56.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:01:56.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:01:56.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:01:56.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:01:56.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:01:56.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:01:56.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:01:56.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:01:56.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:01:56.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:01:56.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-09-16T17:01:56.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:01:56.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-16T17:01:57.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:01:57.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:01:57.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:01:57.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:01:57.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:01:57.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:01:57.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:01:57.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:01:58.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:01:58.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:01:58.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:01:58.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:01:58.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:01:58.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:01:58.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:01:58.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:01:58.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:01:58.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:01:58.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:01:58.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:01:58.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:01:59.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:01:59.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:01:59.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 13 )) 2024-09-16T17:01:59.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:01:59.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-16T17:02:00.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:02:00.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:02:00.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:02:00.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:02:00.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:02:00.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:02:00.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:02:00.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:02:01.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:02:01.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:02:01.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:02:01.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:02:01.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:02:01.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:02:01.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:02:01.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:02:01.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:02:01.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:02:01.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:02:01.875 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:02:01.875 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:02:02.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:02:02.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:02:02.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 13 )) 2024-09-16T17:02:02.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:02:02.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-16T17:02:05.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:02:05.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:02:05.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:02:05.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:02:05.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:02:05.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:02:05.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:02:05.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:02:06.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:02:06.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:02:06.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:02:06.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:02:06.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:02:06.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:02:06.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:02:06.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:02:06.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:02:06.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:02:06.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:02:06.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:02:06.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:02:07.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:02:07.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:02:07.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 13 )) 2024-09-16T17:02:07.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:02:07.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-16T17:02:13.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:02:13.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:02:13.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:02:13.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:02:13.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:02:13.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:02:13.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:02:13.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:02:14.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:02:14.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:02:14.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:02:14.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:02:14.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:02:14.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:02:14.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:02:14.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:02:14.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:02:14.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:02:14.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:02:14.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:02:14.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:02:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:02:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:02:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 13 )) 2024-09-16T17:02:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:02:15.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 12.8 2024-09-16T17:02:27.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:02:27.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:02:27.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:02:27.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:02:27.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:02:27.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:02:27.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:02:27.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:02:28.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:02:28.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:02:28.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:02:28.424 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:02:28.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:02:28.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:02:28.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:02:28.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:02:28.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:02:28.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:02:28.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:02:28.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:02:28.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:02:29.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:02:29.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:02:29.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 13 )) 2024-09-16T17:02:29.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:02:29.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-16T17:02:44.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:02:44.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:02:44.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:02:44.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:02:44.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:02:44.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:02:44.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:02:44.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:02:44.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:02:44.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:02:44.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:02:44.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:02:45.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:02:45.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:02:45.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:02:45.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:02:45.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:02:45.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:02:45.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:02:45.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:02:45.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:02:46.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:02:46.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:02:46.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 9 >= 13 )) 2024-09-16T17:02:46.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:02:46.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-16T17:03:01.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:03:01.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:03:01.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:03:01.042 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:03:01.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:03:01.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:03:01.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:03:01.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:03:01.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:03:01.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:03:01.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:03:01.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:03:02.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:03:02.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:03:02.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:03:02.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:03:02.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:03:02.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:03:02.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:03:02.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:03:02.006 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:03:02.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:03:02.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:03:02.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 10 >= 13 )) 2024-09-16T17:03:02.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:03:02.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-16T17:03:17.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:03:17.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:03:17.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:03:17.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:03:17.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:03:17.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:03:17.552 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:03:17.552 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:03:17.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:03:17.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:03:17.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:03:17.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:03:18.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:03:18.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:03:18.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:03:18.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:03:18.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:03:18.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:03:18.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:03:18.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:03:18.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:03:19.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:03:19.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:03:19.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 11 >= 13 )) 2024-09-16T17:03:19.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:03:19.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-16T17:03:34.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:03:34.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:03:34.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:03:34.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:03:34.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:03:34.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:03:34.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:03:34.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:03:34.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:03:34.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:03:34.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:03:34.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:03:35.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:03:35.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:03:35.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:03:35.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:03:35.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:03:35.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:03:35.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:03:35.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:03:35.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:03:35.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:03:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:03:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 12 >= 13 )) 2024-09-16T17:03:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:03:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 4.5 2024-09-16T17:03:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:03:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:03:40.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:03:40.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:03:40.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:03:40.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:03:40.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:03:40.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:03:40.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:03:40.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:03:40.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:03:40.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:03:41.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:03:41.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:03:41.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:03:41.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:03:41.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:03:41.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:03:41.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:03:41.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:03:41.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:03:41.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:03:41.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:03:41.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 13 >= 13 )) 2024-09-16T17:03:41.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T17:03:42.175 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T17:03:42.175 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "1ca7954d-bc0e-4b69-9628-860fd50073f3", 2024-09-16T17:03:42.175 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "18.2.4-845-ge4e227b3", 2024-09-16T17:03:42.175 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "e4e227b37858f5109138494e01e5ba1ba7b8ffe8", 2024-09-16T17:03:42.175 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-16T17:03:42.172925+0000", 2024-09-16T17:03:42.175 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-16T17:03:42.175 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_ERR", 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: "OBJECT_UNFOUND": { 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1/1 objects unfound (100.000%)", 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 2.0 has 1 unfound objects" 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.176 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.177 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T17:03:42.177 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.177 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DAMAGED": { 2024-09-16T17:03:42.177 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_ERR", 2024-09-16T17:03:42.177 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T17:03:42.177 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Possible data damage: 1 pg recovery_unfound", 2024-09-16T17:03:42.177 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T17:03:42.177 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.177 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T17:03:42.177 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.177 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 3/3 objects degraded (100.000%), 1 pg degraded", 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.178 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "2 pool(s) do not have an application enabled", 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 2 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'foo'" 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'pool-rep'" 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: "SLOW_OPS": { 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 slow ops, oldest one blocked for 120 sec, osd.3 has slow ops", 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-16T17:03:42.180 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "002d35f1-7609-4332-991c-0467e7c6bcdf", 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T16:58:19.271506Z", 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T16:58:19.271506Z", 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 18, 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "reef", 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-16T17:03:42.181 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: "reef" 2024-09-16T17:03:42.182 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7140", 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.183 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7140/0", 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7140/0", 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 39, 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "002d35f1-7609-4332-991c-0467e7c6bcdf", 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-16T16:58:20.264958+0000", 2024-09-16T17:03:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T17:01:01.309950+0000", 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-16T17:00:51.578273+0000", 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-16T17:00:13.590136+0000", 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5799936, 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 9, 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T17:03:42.185 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 2, 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 4, 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "luminous", 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "reef", 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "foo", 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-16T16:58:21.323803+0000", 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-16T17:03:42.186 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 8, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 8, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 8, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 8, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 8, 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-16T17:03:42.187 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "8", 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-16T17:03:42.188 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-16T17:03:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-16T17:03:42.190 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 1.5, 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 1.5, 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 1, 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 1.5, 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 1.5, 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 2, 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "pool-rep", 2024-09-16T17:03:42.191 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-16T17:00:59.690904+0000", 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 1, 2024-09-16T17:03:42.192 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 1, 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 1, 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 1, 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "39", 2024-09-16T17:03:42.193 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-16T17:03:42.194 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T17:03:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 4, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 4, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 0.75, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 3, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 3, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-16T17:03:42.196 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "ac96af98-3977-4cc7-bba7-a0097a0bcf7f", 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 6, 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 34, 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 0, 2024-09-16T17:03:42.197 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 75325826 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 75325826 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T17:03:42.198 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 75325826 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 75325826 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.199 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 75325826 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 75325826 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 75325826 2024-09-16T17:03:42.200 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 75325826 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/75325826", 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/75325826", 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/75325826", 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/75325826", 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T17:03:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "9249db3a-ff13-4802-a94b-4380195eac20", 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 12, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 34, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 0, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T17:03:42.202 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1437558630 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1437558630 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.203 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1437558630 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1437558630 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1437558630 2024-09-16T17:03:42.204 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1437558630 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1437558630 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.205 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1437558630 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/1437558630", 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/1437558630", 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1437558630", 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1437558630", 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.206 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "9ca493da-48d8-4bf8-9217-8a926d133a52", 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 19, 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 34, 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 0, 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6818", 2024-09-16T17:03:42.207 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3506879140 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6819", 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3506879140 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6820", 2024-09-16T17:03:42.208 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3506879140 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6821", 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3506879140 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6824", 2024-09-16T17:03:42.209 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3506879140 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6825", 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3506879140 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6822", 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3506879140 2024-09-16T17:03:42.210 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6823", 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3506879140 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6819/3506879140", 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6821/3506879140", 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6825/3506879140", 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6823/3506879140", 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.211 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 3, 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "ee558f23-218a-4788-ac38-ee3188b7e8d9", 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 32, 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 37, 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 0, 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.212 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6826", 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1114445185 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6827", 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1114445185 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6828", 2024-09-16T17:03:42.213 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1114445185 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6829", 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1114445185 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6832", 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1114445185 2024-09-16T17:03:42.214 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6833", 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1114445185 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6830", 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1114445185 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.215 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6831", 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1114445185 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6827/1114445185", 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6829/1114445185", 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6833/1114445185", 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6831/1114445185", 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.216 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "0.000000", 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T16:58:55.551887+0000", 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "0.000000", 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T17:03:42.217 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T16:59:33.220003+0000", 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "0.000000", 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T17:03:42.218 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T17:00:11.432026+0000", 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 3, 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "0.000000", 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540138322906710015, 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-16T17:00:51.127319+0000", 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [ 2024-09-16T17:03:42.219 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: "pgid": "1.3", 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: "mappings": [ 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: "from": 0, 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: "to": 3 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: "pgid": "1.4", 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: "mappings": [ 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: "from": 0, 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: "to": 2 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.220 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.221 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-16T17:03:42.222 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/75325826,v1:127.0.0.1:6805/75325826]", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/0/block", 2024-09-16T17:03:42.223 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T16:58:27.183690Z", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T17:03:42.224 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/75325826,v1:127.0.0.1:6803/75325826]", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/75325826,v1:127.0.0.1:6809/75325826]", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/75325826,v1:127.0.0.1:6807/75325826]", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T17:03:42.225 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/osd-rep-recov-eio.sh/0", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/1437558630,v1:127.0.0.1:6813/1437558630]", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T17:03:42.226 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/1/block", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T17:03:42.227 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T16:59:03.483276Z", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/1437558630,v1:127.0.0.1:6811/1437558630]", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1437558630,v1:127.0.0.1:6817/1437558630]", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1437558630,v1:127.0.0.1:6815/1437558630]", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T17:03:42.228 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/osd-rep-recov-eio.sh/1", 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6820/3506879140,v1:127.0.0.1:6821/3506879140]", 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T17:03:42.229 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/2/block", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T17:03:42.230 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T17:03:42.231 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T17:03:42.231 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T17:03:42.231 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T17:03:42.231 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T17:03:42.231 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T16:59:41.352682Z", 2024-09-16T17:03:42.231 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T17:03:42.231 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T17:03:42.231 INFO:tasks.workunit.client.0.smithi039.stderr:report 2512402260 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6818/3506879140,v1:127.0.0.1:6819/3506879140]", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/3506879140,v1:127.0.0.1:6825/3506879140]", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/3506879140,v1:127.0.0.1:6823/3506879140]", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T17:03:42.232 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/osd-rep-recov-eio.sh/2", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 3, 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6828/1114445185,v1:127.0.0.1:6829/1114445185]", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-16T17:03:42.233 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/3/block", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "reef", 2024-09-16T17:03:42.234 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-16T17:00:19.742686Z", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6826/1114445185,v1:127.0.0.1:6827/1114445185]", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6832/1114445185,v1:127.0.0.1:6833/1114445185]", 2024-09-16T17:03:42.235 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6830/1114445185,v1:127.0.0.1:6831/1114445185]", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746860", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/osd-rep-recov-eio.sh/3", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.236 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [ 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: "floor": 4294967295 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 7 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.237 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 13 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 20 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 3, 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 33 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 39, 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-16T17:03:42.238 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.239 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 3, 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.3", 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.240 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-16T17:03:42.241 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-16T17:03:42.242 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 25608, 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T17:03:42.243 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 25608, 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 25608, 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T17:03:42.244 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 25608, 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 25608, 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.245 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 3, 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T17:03:42.246 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 3 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 25608, 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-16T17:03:42.247 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 3, 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-16T17:03:42.248 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 3 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-16T17:03:42.249 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-16T17:03:42.250 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 0, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0 2024-09-16T17:03:42.251 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T17:03:42.252 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2" 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-16T17:03:42.253 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 6, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 4 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 9, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 9, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 4, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 4096, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 1, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 3, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 1, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 3, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T17:03:42.254 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 1, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 1, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 3, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 12, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 1, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 4, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 3, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 12288, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T17:03:42.255 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 3 2024-09-16T17:03:42.256 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.257 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 1, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 1, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 27, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 27, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 27, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 4, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 4, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 4, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 419430400, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 108600, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 1292, 2024-09-16T17:03:42.258 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 6, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 107257, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 419321800, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 429496729600, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 429385523200, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1323008, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 451352, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 6374, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 109831962 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T17:03:42.259 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 4, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [ 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.260 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: 1 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 262144 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.261 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 27, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 4, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 4, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 4, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 419430400, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 108600, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 1292, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 6, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 107257, 2024-09-16T17:03:42.262 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 419321800, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 429496729600, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 429385523200, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1323008, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 451352, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 6374, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 109831962 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-16T17:03:42.263 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 4, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [ 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.264 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: 0, 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: 1 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 262144 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.265 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 2, 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 4096, 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 1, 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 3, 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 1, 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 3, 2024-09-16T17:03:42.266 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 1, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 1, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 3, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 12, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 1, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 4, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 3, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 12288, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T17:03:42.267 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T17:03:42.268 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 3 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 12288, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 12288, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 1, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 1, 2024-09-16T17:03:42.269 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 3, 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 3, 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 8, 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 0, 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 0, 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 0, 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-16T17:03:42.270 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-16T17:03:42.271 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-16T17:03:42.272 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 0, 2024-09-16T17:03:42.273 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 0, 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 24, 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 24, 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 4 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 25769803852 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 51539607622 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T17:03:42.274 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 81604378685 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 3, 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 137438953530 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+recovery_unfound+degraded", 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 1 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+clean", 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 8 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.275 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 3, 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 4, 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 8, 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 6, 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.276 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 3, 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 6, 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 7, 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-16T17:03:42.277 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 2, 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 6, 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-16T17:01:51.933364+0000", 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 150, 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 652, 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-16T17:03:42.278 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 252, 2024-09-16T17:03:42.279 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 847, 2024-09-16T17:03:42.279 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-16T17:03:42.279 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-16T17:03:42.279 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-16T17:03:42.279 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T17:03:42.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T17:03:42.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: ceph pg 2.0 query 2024-09-16T17:03:42.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-09-16T17:03:42.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: COUNT=3 2024-09-16T17:03:42.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:161: rados_get_data: test 3 = 3 2024-09-16T17:03:42.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:162: rados_get_data: flush_pg_stats 2024-09-16T17:03:42.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:03:42.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:03:42.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:03:42.791 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:03:42.791 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:03:42.791 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:03:42.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:03:42.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:03:42.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:03:42.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803855 2024-09-16T17:03:42.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803855 2024-09-16T17:03:42.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803855' 2024-09-16T17:03:42.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:03:42.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:03:43.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607624 2024-09-16T17:03:43.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607624 2024-09-16T17:03:43.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803855 1-51539607624' 2024-09-16T17:03:43.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:03:43.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:03:43.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378688 2024-09-16T17:03:43.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378688 2024-09-16T17:03:43.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803855 1-51539607624 2-81604378688' 2024-09-16T17:03:43.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:03:43.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:03:43.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=137438953532 2024-09-16T17:03:43.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 137438953532 2024-09-16T17:03:43.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803855 1-51539607624 2-81604378688 3-137438953532' 2024-09-16T17:03:43.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:03:43.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803855 2024-09-16T17:03:43.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:03:43.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:03:43.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803855 2024-09-16T17:03:43.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:03:43.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803855 2024-09-16T17:03:43.403 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803855 2024-09-16T17:03:43.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803855' 2024-09-16T17:03:43.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:03:43.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803852 -lt 25769803855 2024-09-16T17:03:43.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:03:44.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:03:44.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:03:45.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803855 -lt 25769803855 2024-09-16T17:03:45.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:03:45.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607624 2024-09-16T17:03:45.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:03:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:03:45.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607624 2024-09-16T17:03:45.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:03:45.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607624 2024-09-16T17:03:45.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607624' 2024-09-16T17:03:45.282 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607624 2024-09-16T17:03:45.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:03:45.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607624 -lt 51539607624 2024-09-16T17:03:45.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:03:45.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378688 2024-09-16T17:03:45.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:03:45.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:03:45.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378688 2024-09-16T17:03:45.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:03:45.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378688 2024-09-16T17:03:45.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378688' 2024-09-16T17:03:45.699 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378688 2024-09-16T17:03:45.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:03:46.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378688 -lt 81604378688 2024-09-16T17:03:46.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:03:46.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-137438953532 2024-09-16T17:03:46.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:03:46.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:03:46.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-137438953532 2024-09-16T17:03:46.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:03:46.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=137438953532 2024-09-16T17:03:46.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 137438953532' 2024-09-16T17:03:46.101 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 137438953532 2024-09-16T17:03:46.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:03:46.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 137438953532 -lt 137438953532 2024-09-16T17:03:46.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: ceph pg dump --format=json-pretty 2024-09-16T17:03:46.504 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-16T17:03:46.903 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-09-16T17:03:46.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: COUNT=4 2024-09-16T17:03:46.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:164: rados_get_data: test 4 = 4 2024-09-16T17:03:46.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:177: TEST_rados_get_with_eio: delete_pool pool-rep 2024-09-16T17:03:46.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-09-16T17:03:46.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-09-16T17:03:47.498 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'pool-rep' does not exist 2024-09-16T17:03:47.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-16T17:03:47.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:03:47.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:03:47.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-16T17:03:47.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:03:47.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:03:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:03:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:03:47.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:03:47.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:03:47.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:03:47.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:03:47.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:03:47.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:03:47.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:03:47.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:03:47.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:03:47.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:03:47.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:03:47.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:03:47.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:03:47.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:03:47.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-16T17:03:47.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:03:47.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:03:47.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:03:47.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383662 2024-09-16T17:03:47.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:03:47.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:03:47.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-16T17:03:47.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-16T17:03:47.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:03:47.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-16T17:03:47.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:03:47.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:03:47.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-16T17:03:47.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:03:47.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:03:47.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:03:47.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:03:47.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:03:47.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:03:47.874 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:03:47.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:03:47.876 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:03:47.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:03:47.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:03:47.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:03:47.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:03:47.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:03:47.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:03:47.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:03:47.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:03:47.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:03:47.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-16T17:03:47.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:03:47.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:03:47.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:03:47.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383662 2024-09-16T17:03:47.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:03:47.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:03:47.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-16T17:03:47.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:03:47.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:03:47.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:03:47.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.383662 2024-09-16T17:03:47.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:03:47.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T17:03:47.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:03:47.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-16T17:03:47.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-16T17:03:47.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:03:47.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:03:47.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:03:47.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:03:47.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-16T17:03:47.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-16T17:03:48.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:03:48.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:03:48.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:03:48.270 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:03:48.270 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:03:48.270 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:03:48.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:03:48.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-16T17:03:48.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:03:48.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:03:48.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:03:48.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:03:48.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:03:48.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:03:48.467 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:03:48.467 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:03:48.467 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:03:48.468 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:03:48.468 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:03:48.468 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:03:48.468 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-mon.a.asok 2024-09-16T17:03:48.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:03:48.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383662/ceph-mon.a.asok config get fsid 2024-09-16T17:03:48.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:03:48.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:03:48.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:03:48.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:03:48.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:03:48.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:03:48.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:03:48.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:03:48.589 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:03:48.589 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:03:48.589 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:03:48.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-mon.a.asok 2024-09-16T17:03:48.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:03:48.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383662/ceph-mon.a.asok config get mon_host 2024-09-16T17:03:48.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-16T17:03:48.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:03:48.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:03:48.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:03:48.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:03:48.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-16T17:03:48.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:03:49.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:03:49.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:03:49.626 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:03:49.627 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:03:49.627 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:03:49.627 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:03:49.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:03:49.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:03:49.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:03:49.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-16T17:03:50.020 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T17:03:50.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_repair_warning td/osd-rep-recov-eio.sh 2024-09-16T17:03:50.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:181: TEST_rados_repair_warning: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:03:50.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: expr 10 + 1 2024-09-16T17:03:50.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: local OBJS=11 2024-09-16T17:03:50.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:184: TEST_rados_repair_warning: setup_osds 4 2024-09-16T17:03:50.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-09-16T17:03:50.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-16T17:03:50.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-16T17:03:50.039 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-09-16T17:03:50.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-09-16T17:03:50.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T17:03:50.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-16T17:03:50.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:03:50.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:03:50.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:03:50.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:03:50.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:03:50.043 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:03:50.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:03:50.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:03:50.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:03:50.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:03:50.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:03:50.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:03:50.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:03:50.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:03:50.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:03:50.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:03:50.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:03:50.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:03:50.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:03:50.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-16T17:03:50.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:03:50.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=48c67675-bd9e-49d5-be75-ec495540aea1 2024-09-16T17:03:50.046 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 48c67675-bd9e-49d5-be75-ec495540aea1 2024-09-16T17:03:50.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 48c67675-bd9e-49d5-be75-ec495540aea1' 2024-09-16T17:03:50.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:03:50.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2ZOhm8d2fAxAAeQktRnySAopMJahX1duriQ== 2024-09-16T17:03:50.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2ZOhm8d2fAxAAeQktRnySAopMJahX1duriQ=="}' 2024-09-16T17:03:50.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 48c67675-bd9e-49d5-be75-ec495540aea1 -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-16T17:03:50.406 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:03:50.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-16T17:03:50.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2ZOhm8d2fAxAAeQktRnySAopMJahX1duriQ== --osd-uuid 48c67675-bd9e-49d5-be75-ec495540aea1 2024-09-16T17:03:50.459 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:03:50.459+0000 7ff6158e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:03:50.464 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:03:50.463+0000 7ff6158e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:03:50.466 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:03:50.463+0000 7ff6158e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:03:50.466 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:03:50.467+0000 7ff6158e07c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T17:03:50.466 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:03:50.467+0000 7ff6158e07c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T17:03:50.467 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:03:50.467+0000 7ff6158e07c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T17:03:50.505 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:03:50.503+0000 7ff6158e07c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-16T17:03:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-16T17:03:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:03:56.704 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:03:56.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:03:56.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:03:57.272 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:03:57.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:03:57.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:03:57.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:03:57.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:03:57.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:03:57.304 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:03:57.303+0000 7fc690e727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:03:57.339 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:03:57.339+0000 7fc690e727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:03:57.341 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:03:57.339+0000 7fc690e727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:03:57.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:03:57.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:03:57.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:03:57.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:03:57.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:03:57.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:03:57.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:03:57.728 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:03:57.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:03:57.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:03:57.908 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:03:57.907+0000 7fc690e727c0 -1 Falling back to public interface 2024-09-16T17:03:58.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:03:59.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:03:59.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:03:59.167 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:03:59.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:03:59.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:03:59.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:03:59.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:00.507 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:00.506+0000 7fc690e727c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:04:00.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:00.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:00.642 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:04:00.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:04:00.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:00.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:01.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:02.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:02.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:02.192 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:04:02.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:04:02.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:02.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:02.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:03.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:03.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:03.642 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:04:03.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:04:03.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:03.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:04.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:05.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:05.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:05.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:04:05.081 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:04:05.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:05.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:05.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:06.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:06.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:06.510 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:04:06.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:04:06.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:06.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:06.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:07.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:07.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:07.937 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:04:07.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:04:07.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:07.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:08.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:09.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:09.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:09.474 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:04:09.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:04:09.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:09.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:09.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:10.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:10.932 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:04:10.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:10.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:04:10.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:10.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:11.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:12.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:12.437 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:04:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:04:12.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:12.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:12.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:13.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:13.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:13.918 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:04:13.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:04:13.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:13.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:14.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:15.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:15.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:15.807 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:04:15.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:04:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:16.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:17.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:17.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:17.952 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:04:17.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:04:17.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:17.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:18.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:19.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:19.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:19.776 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:04:19.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:04:19.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:19.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:20.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:21.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:21.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:21.384 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:04:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:04:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:22.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:23.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:23.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:23.395 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:04:23.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:04:23.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:23.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:23.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:24.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:24.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:24.977 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:04:24.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:04:24.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:24.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:04:25.543 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3610304365,v1:127.0.0.1:6803/3610304365] [v2:127.0.0.1:6804/3610304365,v1:127.0.0.1:6805/3610304365] exists,up 48c67675-bd9e-49d5-be75-ec495540aea1 2024-09-16T17:04:25.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:04:25.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:04:25.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:04:25.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T17:04:25.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-16T17:04:25.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:04:25.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:04:25.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T17:04:25.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:04:25.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-16T17:04:25.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:04:25.547 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:04:25.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-16T17:04:25.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:04:25.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7a276f0-7ad2-4819-859e-89b60350eb87 2024-09-16T17:04:25.549 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 e7a276f0-7ad2-4819-859e-89b60350eb87 2024-09-16T17:04:25.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e7a276f0-7ad2-4819-859e-89b60350eb87' 2024-09-16T17:04:25.550 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:04:25.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZZehm4QjjIRAA0Bbh1j//Ffq+D3dKBnldFg== 2024-09-16T17:04:25.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZZehm4QjjIRAA0Bbh1j//Ffq+D3dKBnldFg=="}' 2024-09-16T17:04:25.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7a276f0-7ad2-4819-859e-89b60350eb87 -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-16T17:04:26.367 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:04:26.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-16T17:04:26.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZZehm4QjjIRAA0Bbh1j//Ffq+D3dKBnldFg== --osd-uuid e7a276f0-7ad2-4819-859e-89b60350eb87 2024-09-16T17:04:26.418 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:26.414+0000 7f8b02a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:04:26.447 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:26.446+0000 7f8b02a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:04:26.449 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:26.446+0000 7f8b02a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:04:26.450 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:26.446+0000 7f8b02a307c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T17:04:26.450 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:26.446+0000 7f8b02a307c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T17:04:26.450 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:26.446+0000 7f8b02a307c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T17:04:26.547 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:26.546+0000 7f8b02a307c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-16T17:04:32.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-16T17:04:32.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:04:32.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T17:04:32.882 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T17:04:32.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:04:33.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T17:04:33.447 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:04:33.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:04:33.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:04:33.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:04:33.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:04:33.477 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:33.474+0000 7fe1e9b2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:04:33.480 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:33.478+0000 7fe1e9b2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:04:33.481 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:33.478+0000 7fe1e9b2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:04:33.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T17:04:33.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:04:33.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:04:33.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:04:33.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:04:33.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:33.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:04:33.930 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:04:33.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:33.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:34.051 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:34.050+0000 7fe1e9b2a7c0 -1 Falling back to public interface 2024-09-16T17:04:34.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:35.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:35.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:35.389 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:04:35.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:04:35.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:35.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:35.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:36.747 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:04:36.746+0000 7fe1e9b2a7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T17:04:36.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:36.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:36.815 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:04:36.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:04:36.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:36.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:37.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:38.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:38.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:38.504 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:04:38.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:04:38.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:38.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:39.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:40.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:40.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:40.147 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:04:40.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:04:40.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:40.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:40.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:41.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:41.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:41.772 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:04:41.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:04:41.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:41.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:42.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:43.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:43.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:43.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:04:43.238 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:04:43.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:43.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:43.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:44.693 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:04:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:04:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:45.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:46.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:46.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:46.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:04:46.274 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:04:46.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:46.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:46.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:47.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:47.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:47.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:04:47.707 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:04:47.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:47.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:48.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:49.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:49.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:49.152 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:04:49.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:04:49.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:49.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:49.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:50.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:50.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:50.723 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:04:50.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:04:50.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:50.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:51.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:52.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:52.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:52.814 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:04:52.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:04:52.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:52.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:54.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:55.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:55.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:55.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:04:55.056 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:04:55.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:55.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:55.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:56.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:56.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:56.866 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:04:56.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:04:56.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:56.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:57.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:04:58.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:04:58.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:04:58.768 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:04:58.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:04:58.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:04:58.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:04:59.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:00.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:00.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:00.788 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:05:00.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:05:00.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:00.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:05:01.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:02.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:02.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:02.464 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:05:02.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:05:02.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:02.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:05:03.059 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2570954851,v1:127.0.0.1:6811/2570954851] [v2:127.0.0.1:6812/2570954851,v1:127.0.0.1:6813/2570954851] exists,up e7a276f0-7ad2-4819-859e-89b60350eb87 2024-09-16T17:05:03.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:05:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:05:03.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-16T17:05:03.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-16T17:05:03.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:05:03.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:05:03.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:05:03.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:05:03.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:05:03.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:05:03.063 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:05:03.063 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:05:03.063 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:05:03.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:05:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:05:03.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-16T17:05:03.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:05:03.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6bd3ca0c-2a37-409e-be38-5289797e68a1 2024-09-16T17:05:03.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6bd3ca0c-2a37-409e-be38-5289797e68a1' 2024-09-16T17:05:03.067 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 6bd3ca0c-2a37-409e-be38-5289797e68a1 2024-09-16T17:05:03.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:05:03.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/ZehmWDPZBBAADhm2aSgO7JQzzsAh102PFg== 2024-09-16T17:05:03.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/ZehmWDPZBBAADhm2aSgO7JQzzsAh102PFg=="}' 2024-09-16T17:05:03.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6bd3ca0c-2a37-409e-be38-5289797e68a1 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-16T17:05:03.691 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:05:03.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-16T17:05:03.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/ZehmWDPZBBAADhm2aSgO7JQzzsAh102PFg== --osd-uuid 6bd3ca0c-2a37-409e-be38-5289797e68a1 2024-09-16T17:05:03.746 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:03.741+0000 7f52628537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:03.760 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:03.753+0000 7f52628537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:03.760 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:03.757+0000 7f52628537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:03.760 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:03.757+0000 7f52628537c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T17:05:03.760 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:03.757+0000 7f52628537c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T17:05:03.760 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:03.757+0000 7f52628537c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T17:05:03.875 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:03.873+0000 7f52628537c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-16T17:05:10.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-16T17:05:10.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:05:10.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T17:05:10.493 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T17:05:10.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:05:11.057 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T17:05:11.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T17:05:11.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:05:11.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:05:11.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:05:11.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:05:11.088 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:11.085+0000 7f45010137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:11.126 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:11.121+0000 7f45010137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:11.128 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:11.125+0000 7f45010137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T17:05:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:05:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:05:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:05:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:05:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:11.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:05:11.558 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:05:11.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:11.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:11.716 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:11.713+0000 7f45010137c0 -1 Falling back to public interface 2024-09-16T17:05:11.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:12.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:12.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:12.998 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:05:12.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:05:12.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:12.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:13.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:14.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:14.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:14.442 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:05:14.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:05:14.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:14.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:14.443 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:14.441+0000 7f45010137c0 -1 osd.2 0 log_to_monitors true 2024-09-16T17:05:15.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:16.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:16.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:16.052 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:05:16.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:05:16.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:16.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:16.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:17.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:17.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:17.584 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:05:17.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:05:17.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:17.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:18.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:19.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:19.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:19.029 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:05:19.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:05:19.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:19.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:19.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:20.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:20.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:20.512 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:05:20.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:05:20.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:20.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:21.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:22.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:22.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:22.110 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:05:22.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:05:22.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:22.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:22.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:23.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:23.524 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:05:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:05:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:23.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:23.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:24.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:24.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:24.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:05:24.960 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:05:24.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:24.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:25.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:26.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:26.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:26.476 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:05:26.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:05:26.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:26.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:27.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:28.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:28.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:28.023 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:05:28.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:05:28.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:28.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:28.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:29.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:29.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:29.582 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:05:29.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:05:29.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:29.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:30.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:31.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:31.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:31.423 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:05:31.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:05:31.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:31.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:32.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:33.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:33.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:33.312 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:05:33.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:05:33.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:33.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:34.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:35.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:35.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:35.403 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:05:35.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:05:35.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:35.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:36.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:37.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:37.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:37.193 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:05:37.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:05:37.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:37.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:37.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:38.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:38.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:38.696 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:05:38.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:05:38.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:38.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:39.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:40.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:40.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:40.212 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:05:40.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T17:05:40.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:40.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:05:40.916 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1762999340,v1:127.0.0.1:6819/1762999340] [v2:127.0.0.1:6820/1762999340,v1:127.0.0.1:6821/1762999340] exists,up 6bd3ca0c-2a37-409e-be38-5289797e68a1 2024-09-16T17:05:40.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:05:40.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-09-16T17:05:40.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-09-16T17:05:40.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:05:40.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:05:40.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:05:40.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:05:40.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:05:40.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:05:40.919 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:05:40.919 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:05:40.919 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:05:40.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:05:40.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:05:40.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:05:40.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:05:40.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:05:40.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:05:40.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:05:40.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:05:40.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:05:40.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:05:40.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:05:40.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:05:40.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:05:40.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-09-16T17:05:40.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:05:40.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=df0ba850-1598-44a3-a83c-84b1c3f4cde6 2024-09-16T17:05:40.922 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 df0ba850-1598-44a3-a83c-84b1c3f4cde6 2024-09-16T17:05:40.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 df0ba850-1598-44a3-a83c-84b1c3f4cde6' 2024-09-16T17:05:40.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:05:40.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBkZehmf/wuOBAAr57QRpDTGvDeC+DK+15zjg== 2024-09-16T17:05:40.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBkZehmf/wuOBAAr57QRpDTGvDeC+DK+15zjg=="}' 2024-09-16T17:05:40.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new df0ba850-1598-44a3-a83c-84b1c3f4cde6 -i td/osd-rep-recov-eio.sh/3/new.json 2024-09-16T17:05:41.741 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:05:41.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-09-16T17:05:41.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkZehmf/wuOBAAr57QRpDTGvDeC+DK+15zjg== --osd-uuid df0ba850-1598-44a3-a83c-84b1c3f4cde6 2024-09-16T17:05:41.793 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:41.788+0000 7fadfc0457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:41.834 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:41.828+0000 7fadfc0457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:41.836 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:41.832+0000 7fadfc0457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:41.836 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:41.832+0000 7fadfc0457c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-16T17:05:41.836 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:41.832+0000 7fadfc0457c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-16T17:05:41.836 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:41.832+0000 7fadfc0457c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-16T17:05:41.907 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:41.904+0000 7fadfc0457c0 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-09-16T17:05:48.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-09-16T17:05:48.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:05:48.306 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T17:05:48.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T17:05:48.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:05:48.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T17:05:48.864 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T17:05:48.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:05:48.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:05:48.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:05:48.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:05:48.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:48.888+0000 7f916f5807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:48.899 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:48.896+0000 7f916f5807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:48.900 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:48.896+0000 7f916f5807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:05:49.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T17:05:49.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:05:49.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T17:05:49.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:05:49.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:05:49.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:49.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:05:49.270 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:05:49.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:49.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:05:49.456 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:49.452+0000 7f916f5807c0 -1 Falling back to public interface 2024-09-16T17:05:49.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:50.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:50.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:50.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:05:50.710 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:05:50.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:50.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:05:51.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:52.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:52.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:52.132 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:05:52.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:05:52.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:52.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:05:52.134 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:05:52.128+0000 7f916f5807c0 -1 osd.3 0 log_to_monitors true 2024-09-16T17:05:52.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:53.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:53.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:53.673 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:05:53.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:05:53.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:53.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:05:54.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:55.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:55.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:55.390 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:05:55.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:05:55.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:55.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:05:56.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:57.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:57.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:57.127 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:05:57.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:05:57.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:57.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:05:58.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:05:59.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:05:59.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:05:59.034 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:05:59.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:05:59.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:05:59.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:05:59.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:00.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:00.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:00.644 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:06:00.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:06:00.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:00.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:01.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:02.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:02.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:02.146 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:06:02.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:06:02.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:02.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:02.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:03.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:03.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:03.601 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:06:03.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:06:03.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:03.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:04.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:05.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:05.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:05.130 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:06:05.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:06:05.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:05.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:05.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:06.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:06.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:06.579 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:06:06.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:06:06.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:06.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:07.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:08.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:08.077 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:06:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:06:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:08.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:09.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:10.050 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:06:10.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:06:10.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:10.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:10.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:11.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:11.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:11.918 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:06:11.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:06:11.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:11.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:12.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:13.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:13.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:13.725 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:06:13.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:06:13.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:13.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:14.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:15.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:15.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:15.392 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:06:15.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:06:15.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:15.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:16.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:17.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:17.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:17.602 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:06:17.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:06:17.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:17.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:18.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:06:19.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:06:19.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:06:19.331 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:06:19.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T17:06:19.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:06:19.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:06:19.822 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2381719045,v1:127.0.0.1:6827/2381719045] [v2:127.0.0.1:6828/2381719045,v1:127.0.0.1:6829/2381719045] exists,up df0ba850-1598-44a3-a83c-84b1c3f4cde6 2024-09-16T17:06:19.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:06:19.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:06:19.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:06:19.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-16T17:06:19.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:06:19.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:06:19.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:06:19.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:06:19.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:06:19.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:06:19.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:06:19.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:06:19.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:06:19.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:06:19.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:06:19.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:06:19.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:06:19.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:06:19.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:06:20.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:06:20.593 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:20.593 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:06:20.593 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:06:20.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:06:20.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:20.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:06:20.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803809 2024-09-16T17:06:20.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803809 2024-09-16T17:06:20.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803809' 2024-09-16T17:06:20.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:20.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:06:21.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607574 2024-09-16T17:06:21.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607574 2024-09-16T17:06:21.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803809 1-51539607574' 2024-09-16T17:06:21.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:21.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:06:21.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378646 2024-09-16T17:06:21.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378646 2024-09-16T17:06:21.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803809 1-51539607574 2-81604378646' 2024-09-16T17:06:21.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:21.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:06:21.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986185 2024-09-16T17:06:21.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986185 2024-09-16T17:06:21.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803809 1-51539607574 2-81604378646 3-133143986185' 2024-09-16T17:06:21.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:21.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803809 2024-09-16T17:06:21.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:21.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:06:21.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803809 2024-09-16T17:06:21.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:21.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803809 2024-09-16T17:06:21.654 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803809 2024-09-16T17:06:21.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-09-16T17:06:21.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:06:22.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803807 -lt 25769803809 2024-09-16T17:06:22.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:06:23.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:06:23.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:06:23.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803810 -lt 25769803809 2024-09-16T17:06:23.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:23.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607574 2024-09-16T17:06:23.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:23.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:06:23.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607574 2024-09-16T17:06:23.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:23.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607574 2024-09-16T17:06:23.485 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607574 2024-09-16T17:06:23.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607574' 2024-09-16T17:06:23.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:06:23.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607575 -lt 51539607574 2024-09-16T17:06:23.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:23.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378646 2024-09-16T17:06:23.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:23.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:06:23.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378646 2024-09-16T17:06:23.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:23.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378646 2024-09-16T17:06:23.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378646' 2024-09-16T17:06:23.908 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378646 2024-09-16T17:06:23.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:06:24.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378646 -lt 81604378646 2024-09-16T17:06:24.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:24.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-133143986185 2024-09-16T17:06:24.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:24.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:06:24.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-133143986185 2024-09-16T17:06:24.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:24.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986185 2024-09-16T17:06:24.351 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 133143986185 2024-09-16T17:06:24.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 133143986185' 2024-09-16T17:06:24.352 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:06:24.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986185 -lt 133143986185 2024-09-16T17:06:24.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:06:24.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:06:24.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:06:25.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T17:06:25.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:06:25.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:06:25.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:06:25.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:06:25.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:06:25.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:06:25.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:06:25.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-09-16T17:06:25.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:06:25.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:06:25.742 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:06:26.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 8 2024-09-16T17:06:26.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 7 '!=' -1 2024-09-16T17:06:26.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T17:06:26.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=7 2024-09-16T17:06:26.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:06:26.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T17:06:26.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:06:26.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:06:26.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:06:26.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:06:26.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:06:26.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:06:26.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:06:26.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:06:26.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:06:26.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:06:26.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:06:26.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:06:27.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T17:06:27.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:06:27.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:06:27.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:186: TEST_rados_repair_warning: local poolname=pool-rep 2024-09-16T17:06:27.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:187: TEST_rados_repair_warning: create_pool pool-rep 1 1 2024-09-16T17:06:27.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-09-16T17:06:27.951 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'pool-rep' already exists 2024-09-16T17:06:27.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T17:06:28.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:188: TEST_rados_repair_warning: wait_for_clean 2024-09-16T17:06:28.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:06:28.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:06:28.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:06:28.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:06:28.972 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:06:28.972 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:06:28.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:06:28.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:06:28.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:06:29.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:06:29.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:06:29.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:06:29.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:06:29.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:06:29.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:06:29.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:06:29.546 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:29.546 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:06:29.546 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:06:29.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:06:29.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:29.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:06:29.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803813 2024-09-16T17:06:29.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803813 2024-09-16T17:06:29.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813' 2024-09-16T17:06:29.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:29.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:06:29.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607578 2024-09-16T17:06:29.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607578 2024-09-16T17:06:29.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813 1-51539607578' 2024-09-16T17:06:29.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:29.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:06:30.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378649 2024-09-16T17:06:30.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378649 2024-09-16T17:06:30.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813 1-51539607578 2-81604378649' 2024-09-16T17:06:30.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:30.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:06:30.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986189 2024-09-16T17:06:30.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986189 2024-09-16T17:06:30.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813 1-51539607578 2-81604378649 3-133143986189' 2024-09-16T17:06:30.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:30.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803813 2024-09-16T17:06:30.274 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:30.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:06:30.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803813 2024-09-16T17:06:30.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:30.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803813 2024-09-16T17:06:30.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-09-16T17:06:30.277 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803813 2024-09-16T17:06:30.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:06:30.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803813 -lt 25769803813 2024-09-16T17:06:30.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:30.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607578 2024-09-16T17:06:30.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:30.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:06:30.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607578 2024-09-16T17:06:30.705 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:30.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607578 2024-09-16T17:06:30.707 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607578 2024-09-16T17:06:30.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607578' 2024-09-16T17:06:30.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:06:31.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607578 -lt 51539607578 2024-09-16T17:06:31.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:31.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378649 2024-09-16T17:06:31.127 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:31.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:06:31.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378649 2024-09-16T17:06:31.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:31.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378649 2024-09-16T17:06:31.131 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378649 2024-09-16T17:06:31.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378649' 2024-09-16T17:06:31.132 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:06:31.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378649 -lt 81604378649 2024-09-16T17:06:31.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:31.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-133143986189 2024-09-16T17:06:31.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:31.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:06:31.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-133143986189 2024-09-16T17:06:31.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:31.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986189 2024-09-16T17:06:31.562 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 133143986189 2024-09-16T17:06:31.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 133143986189' 2024-09-16T17:06:31.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:06:31.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986189 -lt 133143986189 2024-09-16T17:06:31.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:06:31.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:06:31.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:06:32.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:06:32.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:06:32.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:06:32.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:06:32.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:06:32.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:06:32.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:06:32.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:06:32.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-16T17:06:32.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:06:32.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:06:32.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:06:33.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-16T17:06:33.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:06:33.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:06:33.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:190: TEST_rados_repair_warning: local poolname=pool-rep 2024-09-16T17:06:33.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:191: TEST_rados_repair_warning: local objbase=obj-warn 2024-09-16T17:06:33.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:192: TEST_rados_repair_warning: local inject=eio 2024-09-16T17:06:33.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: seq 1 11 2024-09-16T17:06:33.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:33.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-1 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:06:33.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-1 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:33.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:33.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:33.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-1 2024-09-16T17:06:33.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:33.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-16T17:06:33.660 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-16T17:06:33.660 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:34.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:34.108 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:34.108 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:34.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:34.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:34.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:34.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:34.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:34.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:34.109 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:34.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:34.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:34.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:34.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:34.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:34.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:34.110 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:34.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:34.111 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:34.111 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:34.111 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:34.112 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:34.112 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:34.112 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:34.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:34.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:34.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:34.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:34.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:34.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:34.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:34.232 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:34.232 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:34.232 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:34.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:34.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:34.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-09-16T17:06:34.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-09-16T17:06:34.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:34.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:34.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-09-16T17:06:34.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:34.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:34.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:34.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:34.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:34.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-09-16T17:06:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:06:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-2 2024-09-16T17:06:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:06:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:34.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:06:34.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:34.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:06:34.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:34.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:06:34.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:06:34.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:34.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:34.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:34.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-2 2024-09-16T17:06:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:34.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:34.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-2 2024-09-16T17:06:34.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:34.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-09-16T17:06:34.579 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:34.579 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-09-16T17:06:34.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:34.991 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:34.991 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:34.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:34.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:34.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:34.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:34.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:34.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:34.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:34.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:34.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:34.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:34.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:34.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:34.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:34.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:34.994 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:34.994 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:34.995 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:34.995 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:34.995 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:34.995 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:34.995 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:34.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:35.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:35.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:35.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:35.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:35.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:35.111 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:35.111 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:35.112 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:35.112 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:35.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:35.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:35.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-09-16T17:06:35.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-09-16T17:06:35.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:35.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-09-16T17:06:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:35.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:35.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:35.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:35.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-3 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:06:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:06:35.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:35.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-3 2024-09-16T17:06:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:35.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-3 2024-09-16T17:06:35.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:35.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-09-16T17:06:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-09-16T17:06:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:35.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:35.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:35.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:35.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:35.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:35.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:35.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:35.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:35.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:35.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:35.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:35.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:35.915 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:35.915 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:35.916 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:35.916 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:35.916 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:35.916 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:35.917 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:35.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:36.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:36.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:36.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:36.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:36.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:36.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:36.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:36.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:36.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:36.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:36.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:36.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-09-16T17:06:36.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-09-16T17:06:36.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:36.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:36.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-09-16T17:06:36.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:36.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:36.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:36.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:36.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:36.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-4 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:06:36.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:36.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:06:36.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:06:36.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:36.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:36.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:36.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:36.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:36.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:36.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:36.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-4 2024-09-16T17:06:36.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:36.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:36.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:36.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:36.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:36.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-4 2024-09-16T17:06:36.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:36.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-09-16T17:06:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-09-16T17:06:36.411 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:36.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:36.869 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:36.869 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:36.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:36.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:36.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:36.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:36.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:36.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:36.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:36.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:36.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:36.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:36.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:36.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:36.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:36.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:36.873 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:36.873 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:36.873 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:36.873 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:36.873 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:36.873 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:36.873 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:36.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:36.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:36.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:36.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:36.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:36.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:36.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:36.994 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:36.994 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:36.994 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:36.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:36.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:36.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-09-16T17:06:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-09-16T17:06:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-09-16T17:06:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:37.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:37.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:37.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:37.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-5 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:06:37.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:06:37.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:37.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:37.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:37.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-5 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-5 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-09-16T17:06:37.326 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-09-16T17:06:37.327 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:37.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:37.752 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:37.752 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:37.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:37.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:37.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:37.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:37.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:37.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:37.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:37.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:37.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:37.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:37.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:37.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:37.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:37.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:37.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:37.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:37.756 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:37.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:37.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:37.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:37.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:37.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:37.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:37.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:37.882 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:37.882 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:37.882 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:37.882 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:37.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:37.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-09-16T17:06:38.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-09-16T17:06:38.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:38.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:38.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-09-16T17:06:38.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:38.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:38.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:38.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:38.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:38.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:38.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:38.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-09-16T17:06:38.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:06:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-6 2024-09-16T17:06:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:06:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:06:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:06:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:06:38.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:06:38.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:38.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:38.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:38.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:38.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:38.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:38.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:38.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-6 2024-09-16T17:06:38.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:38.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:38.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:38.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:38.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:38.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-6 2024-09-16T17:06:38.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:38.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-09-16T17:06:38.243 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-09-16T17:06:38.243 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:38.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:38.690 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:38.690 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:38.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:38.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:38.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:38.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:38.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:38.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:38.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:38.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:38.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:38.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:38.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:38.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:38.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:38.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:38.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:38.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:38.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:38.693 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:38.693 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:38.693 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:38.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:38.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:38.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:38.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:38.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:38.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:38.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:38.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:38.815 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:38.815 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:38.815 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:38.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:38.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:38.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-09-16T17:06:38.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-09-16T17:06:38.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:38.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:38.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-09-16T17:06:38.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:38.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:38.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:38.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:38.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:38.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:38.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-7 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:06:38.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:06:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-7 2024-09-16T17:06:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:39.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-7 2024-09-16T17:06:39.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:39.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-09-16T17:06:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-09-16T17:06:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:39.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:39.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:39.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:39.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:39.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:39.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:39.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:39.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:39.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:39.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:39.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:39.585 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:39.586 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:39.586 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:39.586 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:39.586 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:39.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:39.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:39.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:39.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:39.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:39.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:39.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:39.711 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:39.711 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:39.711 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:39.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:39.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:39.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-09-16T17:06:39.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-09-16T17:06:39.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:39.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:39.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-09-16T17:06:39.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:39.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:39.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:39.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:39.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:39.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:39.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:39.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-8 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:06:39.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:06:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:40.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-8 2024-09-16T17:06:40.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:40.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:40.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:40.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:40.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:40.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-8 2024-09-16T17:06:40.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:40.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-09-16T17:06:40.083 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-09-16T17:06:40.083 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:40.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:40.516 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:40.516 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:40.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:40.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:40.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:40.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:40.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:40.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:40.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:40.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:40.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:40.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:40.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:40.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:40.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:40.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:40.519 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:40.519 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:40.519 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:40.519 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:40.519 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:40.519 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:40.520 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:40.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:40.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:40.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:40.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:40.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:40.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:40.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:40.642 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:40.642 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:40.643 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:40.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:40.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:40.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-09-16T17:06:40.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-09-16T17:06:40.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:40.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:40.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-09-16T17:06:40.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:40.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:40.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:40.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:40.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:40.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:40.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:40.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-09-16T17:06:40.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:40.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:06:40.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-9 2024-09-16T17:06:40.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:40.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:06:40.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:40.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:06:40.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:40.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:06:40.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:40.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:06:40.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:06:40.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:40.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-9 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-9 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:40.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-09-16T17:06:40.987 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-09-16T17:06:40.987 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:41.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:41.434 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:41.434 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:41.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:41.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:41.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:41.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:41.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:41.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:41.435 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:41.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:41.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:41.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:41.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:41.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:41.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:41.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:41.437 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:41.437 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:41.437 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:41.437 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:41.438 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:41.438 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:41.438 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:41.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:41.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:41.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:41.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:41.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:41.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:41.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:41.562 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:41.562 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:41.562 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:41.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:41.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:41.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-09-16T17:06:41.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-09-16T17:06:41.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:41.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:41.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-09-16T17:06:41.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:41.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:41.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:41.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:41.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:41.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:41.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:41.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-09-16T17:06:41.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:41.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:06:41.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-10 2024-09-16T17:06:41.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:41.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:06:41.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:41.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:06:41.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:41.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:06:41.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:41.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:06:41.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:06:41.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:41.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:41.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:41.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:41.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:41.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:41.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:41.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-10 2024-09-16T17:06:41.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:41.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:41.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:41.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:41.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:41.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-10 2024-09-16T17:06:41.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:41.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-09-16T17:06:41.880 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:41.880 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-09-16T17:06:42.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:42.321 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:42.322 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:42.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:42.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:42.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:42.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:42.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:42.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:42.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:42.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:42.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:42.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:42.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:42.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:42.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:42.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:42.324 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:42.324 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:42.324 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:42.325 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:42.325 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:42.325 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:42.325 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:42.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:42.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:42.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:42.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:42.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:42.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:42.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:42.441 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:42.441 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:42.441 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:42.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:42.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:42.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-09-16T17:06:42.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-09-16T17:06:42.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:42.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:42.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-09-16T17:06:42.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:42.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:42.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:42.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:42.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:42.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:42.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:42.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-09-16T17:06:42.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:42.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-16T17:06:42.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-11 2024-09-16T17:06:42.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:42.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:06:42.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:42.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:06:42.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:42.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:06:42.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:06:42.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:06:42.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:06:42.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:42.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:42.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:42.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:42.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:42.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:42.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:42.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-11 2024-09-16T17:06:42.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:42.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:42.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:42.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:42.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:42.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-11 2024-09-16T17:06:42.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:42.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-09-16T17:06:42.825 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-09-16T17:06:42.826 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:43.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:43.248 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:43.248 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:43.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:43.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:43.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:43.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:43.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:43.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:43.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:43.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:43.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:43.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:43.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:43.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:43.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:43.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:43.251 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:43.252 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:43.252 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:43.252 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:43.252 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:43.252 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:43.253 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:43.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:43.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:43.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:43.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:43.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:43.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:43.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:43.374 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:43.374 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:43.374 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:43.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:43.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:43.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-09-16T17:06:43.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-09-16T17:06:43.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:43.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:43.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-09-16T17:06:43.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:43.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:43.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:43.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:43.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:43.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:43.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: get_pg pool-rep obj-warn-1 2024-09-16T17:06:43.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-09-16T17:06:43.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-warn-1 2024-09-16T17:06:43.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-warn-1 2024-09-16T17:06:43.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T17:06:43.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: local pgid=2.0 2024-09-16T17:06:43.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: get_osds pool-rep obj-warn-1 2024-09-16T17:06:43.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:43.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-16T17:06:43.931 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-16T17:06:43.931 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: object_osds=('3' '1' '0') 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: local object_osds 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:203: TEST_rados_repair_warning: local primary=3 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:204: TEST_rados_repair_warning: local bad_peer=1 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:206: TEST_rados_repair_warning: wait_for_clean 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:06:44.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:06:44.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:06:44.367 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:06:44.367 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:06:44.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:06:44.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:06:44.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:06:44.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:06:44.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:06:44.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:06:44.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:06:44.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:06:44.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:06:44.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:06:44.935 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:44.935 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:06:44.935 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:06:44.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:06:44.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:44.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:06:45.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803818 2024-09-16T17:06:45.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803818 2024-09-16T17:06:45.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803818' 2024-09-16T17:06:45.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:45.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:06:45.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607583 2024-09-16T17:06:45.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607583 2024-09-16T17:06:45.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803818 1-51539607583' 2024-09-16T17:06:45.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:45.271 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:06:45.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378654 2024-09-16T17:06:45.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378654 2024-09-16T17:06:45.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803818 1-51539607583 2-81604378654' 2024-09-16T17:06:45.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:45.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:06:45.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986194 2024-09-16T17:06:45.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986194 2024-09-16T17:06:45.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803818 1-51539607583 2-81604378654 3-133143986194' 2024-09-16T17:06:45.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:45.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803818 2024-09-16T17:06:45.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:45.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:06:45.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803818 2024-09-16T17:06:45.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:45.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803818 2024-09-16T17:06:45.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803818' 2024-09-16T17:06:45.562 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803818 2024-09-16T17:06:45.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:06:45.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803816 -lt 25769803818 2024-09-16T17:06:45.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:06:46.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:06:46.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:06:47.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803819 -lt 25769803818 2024-09-16T17:06:47.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:47.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:47.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607583 2024-09-16T17:06:47.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:06:47.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607583 2024-09-16T17:06:47.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:47.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607583 2024-09-16T17:06:47.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607583' 2024-09-16T17:06:47.444 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607583 2024-09-16T17:06:47.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:06:47.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607583 -lt 51539607583 2024-09-16T17:06:47.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:47.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378654 2024-09-16T17:06:47.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:47.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:06:47.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378654 2024-09-16T17:06:47.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:47.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378654 2024-09-16T17:06:47.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378654' 2024-09-16T17:06:47.898 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378654 2024-09-16T17:06:47.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:06:48.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378655 -lt 81604378654 2024-09-16T17:06:48.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:48.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-133143986194 2024-09-16T17:06:48.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:48.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:06:48.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-133143986194 2024-09-16T17:06:48.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:48.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986194 2024-09-16T17:06:48.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 133143986194' 2024-09-16T17:06:48.331 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 133143986194 2024-09-16T17:06:48.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:06:48.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986194 -lt 133143986194 2024-09-16T17:06:48.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:06:48.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:06:48.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:06:49.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:06:49.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:06:49.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:06:49.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:06:49.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:06:49.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:06:49.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:06:49.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:06:49.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-16T17:06:49.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:06:49.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:06:49.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:06:50.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-16T17:06:50.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:06:50.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:06:50.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: ceph pg 2.0 query 2024-09-16T17:06:50.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-09-16T17:06:50.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: COUNT=11 2024-09-16T17:06:50.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:208: TEST_rados_repair_warning: test 11 = 11 2024-09-16T17:06:50.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:209: TEST_rados_repair_warning: flush_pg_stats 2024-09-16T17:06:50.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:06:50.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:06:50.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:06:50.937 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:50.937 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:06:50.937 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:06:50.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:06:50.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:50.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:06:51.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803822 2024-09-16T17:06:51.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803822 2024-09-16T17:06:51.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822' 2024-09-16T17:06:51.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:51.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:06:51.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607586 2024-09-16T17:06:51.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607586 2024-09-16T17:06:51.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822 1-51539607586' 2024-09-16T17:06:51.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:51.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:06:51.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378658 2024-09-16T17:06:51.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378658 2024-09-16T17:06:51.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822 1-51539607586 2-81604378658' 2024-09-16T17:06:51.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:06:51.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:06:51.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986197 2024-09-16T17:06:51.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986197 2024-09-16T17:06:51.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822 1-51539607586 2-81604378658 3-133143986197' 2024-09-16T17:06:51.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:51.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803822 2024-09-16T17:06:51.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:51.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:06:51.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803822 2024-09-16T17:06:51.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:51.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803822 2024-09-16T17:06:51.559 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803822 2024-09-16T17:06:51.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803822' 2024-09-16T17:06:51.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:06:51.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803819 -lt 25769803822 2024-09-16T17:06:51.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:06:52.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:06:52.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:06:53.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803822 -lt 25769803822 2024-09-16T17:06:53.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:53.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607586 2024-09-16T17:06:53.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:53.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:06:53.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607586 2024-09-16T17:06:53.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:53.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607586 2024-09-16T17:06:53.444 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607586 2024-09-16T17:06:53.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607586' 2024-09-16T17:06:53.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:06:53.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607587 -lt 51539607586 2024-09-16T17:06:53.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:53.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:53.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378658 2024-09-16T17:06:53.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:06:53.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378658 2024-09-16T17:06:53.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:53.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378658 2024-09-16T17:06:53.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378658' 2024-09-16T17:06:53.911 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378658 2024-09-16T17:06:53.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:06:54.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378658 -lt 81604378658 2024-09-16T17:06:54.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:06:54.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-133143986197 2024-09-16T17:06:54.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:06:54.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:06:54.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-133143986197 2024-09-16T17:06:54.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:06:54.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986197 2024-09-16T17:06:54.343 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 133143986197 2024-09-16T17:06:54.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 133143986197' 2024-09-16T17:06:54.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:06:54.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986197 -lt 133143986197 2024-09-16T17:06:54.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-09-16T17:06:54.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-16T17:06:55.182 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-09-16T17:06:55.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: COUNT=11 2024-09-16T17:06:55.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:211: TEST_rados_repair_warning: test 11 = 11 2024-09-16T17:06:55.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: ceph health 2024-09-16T17:06:55.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-09-16T17:06:55.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: ceph health detail 2024-09-16T17:06:55.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: grep -q 'osd.3 had 11 reads repaired' 2024-09-16T17:06:56.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:216: TEST_rados_repair_warning: ceph health mute OSD_TOO_MANY_REPAIRS 2024-09-16T17:06:57.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:217: TEST_rados_repair_warning: set -o pipefail 2024-09-16T17:06:57.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: ceph health 2024-09-16T17:06:57.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-09-16T17:06:57.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:220: TEST_rados_repair_warning: set +o pipefail 2024-09-16T17:06:57.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: seq 1 11 2024-09-16T17:06:57.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:57.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:57.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:57.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:57.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:57.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:57.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:57.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:57.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-1 2024-09-16T17:06:57.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:57.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:57.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:57.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:57.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:57.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-1 2024-09-16T17:06:57.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:57.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-16T17:06:57.577 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-16T17:06:57.578 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:58.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:58.009 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:58.009 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:58.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:58.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:58.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:58.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:58.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:58.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:58.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:58.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:58.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:58.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:58.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:58.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:58.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:58.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:58.012 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:58.012 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:58.012 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:58.012 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:58.012 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:58.013 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:58.013 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:58.013 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:58.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:58.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:58.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:58.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:58.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:58.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:58.136 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:58.137 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:58.137 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:58.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:58.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:58.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-09-16T17:06:58.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 1 2024-09-16T17:06:58.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-1 2024-09-16T17:06:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:06:58.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:58.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-1 2024-09-16T17:06:58.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:58.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-16T17:06:58.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:58.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-16T17:06:58.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:58.686 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:58.686 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:58.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:58.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:58.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:58.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:06:58.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:58.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:58.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:06:58.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:58.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:06:58.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:58.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:06:58.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:58.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:58.688 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:58.689 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:06:58.689 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:06:58.689 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:06:58.690 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:58.690 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:58.690 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:58.690 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:06:58.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:58.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:58.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:58.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:58.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:06:58.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:06:58.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:06:58.811 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:58.811 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:58.811 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:58.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:06:58.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:58.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-warn-1 2024-09-16T17:06:58.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:06:58.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:58.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:58.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:06:58.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:06:58.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:58.935 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:58.935 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:58.935 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:58.935 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:58.935 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:58.936 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:58.936 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:58.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:06:59.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:59.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-09-16T17:06:59.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:59.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:06:59.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-09-16T17:06:59.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:06:59.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:06:59.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:06:59.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:59.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:59.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-2 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:06:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:59.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-2 2024-09-16T17:06:59.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:59.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-09-16T17:06:59.178 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:06:59.178 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-09-16T17:06:59.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:06:59.595 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:06:59.595 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:06:59.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:06:59.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:06:59.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:06:59.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:06:59.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:06:59.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:06:59.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:06:59.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:06:59.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:06:59.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:06:59.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:06:59.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:06:59.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:06:59.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:06:59.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:06:59.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:59.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:59.598 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:59.599 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:59.599 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:59.599 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:59.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:06:59.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:06:59.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:06:59.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:06:59.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:06:59.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:06:59.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:06:59.724 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:06:59.724 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:06:59.724 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:06:59.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:06:59.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:06:59.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-09-16T17:06:59.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 1 2024-09-16T17:06:59.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:06:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:06:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:06:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:06:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:06:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:06:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-2 2024-09-16T17:06:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:06:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:06:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:06:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:06:59.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:06:59.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-2 2024-09-16T17:06:59.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:06:59.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-09-16T17:06:59.838 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-09-16T17:06:59.838 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:00.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:00.255 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:00.255 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:00.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:00.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:00.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:00.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:07:00.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:00.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:00.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:07:00.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:00.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:07:00.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:00.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:07:00.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:00.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:00.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:07:00.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:00.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:00.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:00.258 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:00.258 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:00.258 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:00.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:00.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:00.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:00.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:00.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:00.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:07:00.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:00.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:00.370 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:00.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-warn-2 2024-09-16T17:07:00.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:07:00.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:00.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:00.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:07:00.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:07:00.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:00.496 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:00.496 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:00.496 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:00.497 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:00.497 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:00.497 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:00.497 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:00.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:07:00.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:00.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-09-16T17:07:00.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:00.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:07:00.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-09-16T17:07:00.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:07:00.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:07:00.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:07:00.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:00.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:00.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-3 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:00.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:00.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:07:00.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:00.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-3 2024-09-16T17:07:00.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:00.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-09-16T17:07:00.667 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-09-16T17:07:00.667 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:01.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:01.145 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:01.145 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:01.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:01.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:01.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:01.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:07:01.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:01.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:01.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:07:01.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:01.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:07:01.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:01.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:01.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:01.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:01.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:01.148 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:01.148 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:01.148 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:01.149 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:01.149 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:01.149 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:01.149 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:01.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:01.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:01.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:01.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:01.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:07:01.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:01.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:01.277 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:01.277 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:01.278 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:01.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:01.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:01.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-09-16T17:07:01.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 1 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-3 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:07:01.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:01.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-3 2024-09-16T17:07:01.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:01.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-09-16T17:07:01.399 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-09-16T17:07:01.399 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:01.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:01.798 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:01.798 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:01.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:01.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:01.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:01.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:07:01.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:01.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:01.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:07:01.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:01.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:07:01.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:01.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:07:01.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:01.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:01.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:01.800 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:07:01.800 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:01.801 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:01.801 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:01.801 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:01.801 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:01.801 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:01.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:01.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:01.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:01.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:01.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:07:01.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:01.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:01.914 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:01.914 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:01.914 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:01.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:01.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:01.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-warn-3 2024-09-16T17:07:02.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:07:02.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:02.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:02.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:07:02.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:07:02.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:02.035 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:02.035 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:02.035 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:02.036 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:02.036 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:02.036 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:02.036 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:02.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:07:02.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:02.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-09-16T17:07:02.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:02.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:07:02.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-09-16T17:07:02.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:07:02.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:07:02.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:07:02.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:02.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:02.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:02.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:07:02.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-4 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:07:02.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:02.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-4 2024-09-16T17:07:02.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:02.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-09-16T17:07:02.217 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-09-16T17:07:02.217 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:02.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:02.632 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:02.632 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:02.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:02.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:02.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:02.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:07:02.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:02.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:02.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:07:02.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:02.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:07:02.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:02.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:02.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:02.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:02.636 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:02.636 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:02.636 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:02.636 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:02.636 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:02.636 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:02.636 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:02.636 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:02.637 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:02.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:02.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:02.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:07:02.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:02.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:02.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:02.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:02.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-09-16T17:07:02.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 1 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-4 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:02.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-4 2024-09-16T17:07:02.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:02.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-09-16T17:07:02.884 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-09-16T17:07:02.884 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:03.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:03.324 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:03.324 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:03.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:03.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:03.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:03.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:07:03.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:03.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:03.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:07:03.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:03.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:07:03.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:03.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:07:03.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:03.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:03.327 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:03.327 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:07:03.327 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:03.327 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:03.328 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:03.328 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:03.328 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:03.328 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:03.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:03.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:03.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:03.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:03.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:07:03.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:03.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:03.445 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:03.445 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:03.445 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:03.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:03.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:03.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-warn-4 2024-09-16T17:07:03.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:07:03.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:03.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:03.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:07:03.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:07:03.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:03.564 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:03.564 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:03.565 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:03.565 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:03.565 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:03.565 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:03.566 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:03.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:07:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-09-16T17:07:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:07:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-09-16T17:07:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:07:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:07:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:07:03.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:03.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:03.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:03.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:07:03.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-09-16T17:07:03.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:03.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-5 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-5 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-09-16T17:07:03.749 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:04.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:04.161 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:04.161 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:04.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:04.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:04.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:04.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:07:04.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:04.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:04.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:07:04.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:04.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:07:04.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:04.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:04.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:04.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:04.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:04.163 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:04.164 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:04.164 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:04.164 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:04.164 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:04.164 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:04.164 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:04.164 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:04.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:04.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:04.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:04.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:07:04.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:04.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:04.281 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:04.281 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:04.281 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:04.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:04.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:04.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-09-16T17:07:04.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 1 2024-09-16T17:07:04.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:04.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-5 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-5 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:04.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-09-16T17:07:04.406 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:04.406 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-09-16T17:07:04.845 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:04.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:07:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:04.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:04.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:07:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:07:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:07:04.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:04.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:04.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:04.848 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:07:04.848 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:04.849 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:04.849 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:04.849 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:04.849 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:04.849 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:04.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:04.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:04.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:04.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:04.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:07:04.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:04.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:04.978 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:04.978 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:04.978 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:04.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:04.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:04.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-warn-5 2024-09-16T17:07:05.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:07:05.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:05.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:05.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:07:05.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:07:05.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:05.100 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:05.100 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:05.100 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:05.101 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:05.101 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:05.101 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:05.101 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:05.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:07:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-09-16T17:07:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:07:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-09-16T17:07:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:07:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:07:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:07:05.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:05.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:05.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:05.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:07:05.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-09-16T17:07:05.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:05.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:05.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:05.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:05.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:05.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:05.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-6 2024-09-16T17:07:05.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:05.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:05.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:05.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:07:05.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:05.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-6 2024-09-16T17:07:05.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:05.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-09-16T17:07:05.282 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-09-16T17:07:05.282 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:05.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:05.684 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:05.684 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:05.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:05.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:05.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:05.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:07:05.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:05.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:05.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:07:05.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:05.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:07:05.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:05.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:05.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:05.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:05.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:05.686 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:05.686 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:05.686 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:05.687 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:05.687 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:05.687 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:05.687 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:05.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:05.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:05.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:05.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:05.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:07:05.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:05.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:05.810 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:05.810 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:05.810 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:05.811 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:05.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:05.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-09-16T17:07:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 1 2024-09-16T17:07:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-6 2024-09-16T17:07:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:07:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:05.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-6 2024-09-16T17:07:05.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:05.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-09-16T17:07:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-09-16T17:07:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:06.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:06.347 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:06.347 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:06.347 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:06.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:06.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:06.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:07:06.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:06.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:06.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:07:06.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:07:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:07:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:06.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:06.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:07:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:06.350 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:06.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:06.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:06.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:06.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:06.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:07:06.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:06.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:06.473 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:06.473 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:06.474 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:06.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:06.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:06.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-warn-6 2024-09-16T17:07:06.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:07:06.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:06.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:06.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:07:06.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:07:06.587 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:06.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:06.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:06.588 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:06.588 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:06.588 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:06.589 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:06.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:06.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:07:06.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:06.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-09-16T17:07:06.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:06.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:07:06.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-09-16T17:07:06.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:07:06.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:07:06.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:07:06.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:06.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:06.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:06.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:07:06.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-7 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:06.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-7 2024-09-16T17:07:06.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:06.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-09-16T17:07:06.749 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-09-16T17:07:06.749 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:07.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:07.152 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:07.152 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:07.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:07.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:07.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:07.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:07:07.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:07.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:07.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:07:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:07:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:07.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:07.154 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:07.155 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:07.155 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:07.155 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:07.155 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:07.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:07.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:07.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:07.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:07.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:07:07.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:07.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:07.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:07.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-09-16T17:07:07.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 1 2024-09-16T17:07:07.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:07.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:07.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:07.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:07.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:07.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:07.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-7 2024-09-16T17:07:07.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:07.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:07.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:07.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:07:07.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:07.392 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-7 2024-09-16T17:07:07.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:07.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-09-16T17:07:07.393 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-09-16T17:07:07.393 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:07.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:07.809 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:07.809 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:07.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:07.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:07.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:07.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:07:07.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:07.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:07.809 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:07:07.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:07.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:07:07.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:07.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:07:07.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:07.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:07.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:07.812 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:07:07.812 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:07.812 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:07.812 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:07.812 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:07.812 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:07.812 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:07.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:07.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:07.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:07.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:07.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:07:07.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:07.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:07.927 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:07.927 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:07.927 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:07.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:07.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:07.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-warn-7 2024-09-16T17:07:08.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:07:08.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:08.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:08.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:07:08.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:07:08.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:08.055 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:08.055 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:08.055 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:08.055 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:08.056 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:08.056 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:08.056 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:08.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:07:08.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:08.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-09-16T17:07:08.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:08.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:07:08.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-09-16T17:07:08.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:07:08.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:07:08.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:07:08.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:08.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:08.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-8 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:07:08.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:08.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-8 2024-09-16T17:07:08.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:08.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-09-16T17:07:08.224 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-09-16T17:07:08.224 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:08.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:08.627 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:08.628 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:08.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:08.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:08.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:08.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:07:08.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:08.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:08.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:07:08.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:08.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:07:08.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:08.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:08.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:08.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:08.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:08.630 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:08.630 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:08.630 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:08.631 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:08.631 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:08.631 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:08.631 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:08.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:08.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:08.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:08.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:07:08.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:08.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:08.747 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:08.747 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:08.747 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:08.748 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:08.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:08.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:08.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-09-16T17:07:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 1 2024-09-16T17:07:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-8 2024-09-16T17:07:08.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:08.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:08.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:08.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:07:08.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:08.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-8 2024-09-16T17:07:08.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:08.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-09-16T17:07:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-09-16T17:07:08.870 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:09.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:09.311 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:09.311 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:09.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:09.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:09.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:09.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:07:09.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:09.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:09.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:07:09.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:09.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:07:09.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:09.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:07:09.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:09.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:09.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:09.314 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:07:09.315 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:09.315 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:09.315 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:09.315 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:09.316 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:09.316 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:09.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:09.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:09.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:09.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:09.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:07:09.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:09.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:09.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:09.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:09.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:09.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:09.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:09.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-warn-8 2024-09-16T17:07:09.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:07:09.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:09.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:09.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:07:09.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:07:09.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:09.562 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:09.562 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:09.562 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:09.563 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:09.563 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:09.563 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:09.563 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:09.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:07:09.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:09.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-09-16T17:07:09.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:09.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:07:09.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-09-16T17:07:09.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:07:09.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:07:09.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:07:09.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:09.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:09.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:09.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:07:09.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-09-16T17:07:09.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:09.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-9 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-9 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-09-16T17:07:09.714 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:10.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:10.119 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:10.119 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:10.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:10.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:10.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:10.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:07:10.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:10.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:10.119 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:07:10.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:10.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:07:10.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:10.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:10.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:10.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:10.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:10.121 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:10.121 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:10.121 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:10.122 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:10.122 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:10.122 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:10.122 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:10.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:10.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:10.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:10.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:10.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:07:10.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:10.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:10.237 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:10.237 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:10.237 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:10.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:10.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:10.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-09-16T17:07:10.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 1 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-9 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:07:10.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:10.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-9 2024-09-16T17:07:10.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:10.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-09-16T17:07:10.351 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-09-16T17:07:10.351 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:10.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:10.804 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:10.804 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:10.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:10.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:10.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:10.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:07:10.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:10.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:10.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:07:10.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:10.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:07:10.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:10.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:07:10.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:10.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:10.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:10.807 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:07:10.807 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:10.807 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:10.807 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:10.808 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:10.808 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:10.808 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:10.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:10.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:10.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:10.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:10.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:07:10.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:10.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:10.935 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:10.935 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:10.935 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:10.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:10.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:10.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-warn-9 2024-09-16T17:07:11.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:07:11.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:11.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:11.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:07:11.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:07:11.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:11.049 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:11.049 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:11.049 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:11.049 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:11.050 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:11.050 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:11.050 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:11.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:07:11.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:11.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-09-16T17:07:11.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:11.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:07:11.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-09-16T17:07:11.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:07:11.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:07:11.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:07:11.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:11.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:11.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:11.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-10 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-10 2024-09-16T17:07:11.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:11.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-09-16T17:07:11.215 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-09-16T17:07:11.215 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:11.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:11.618 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:11.618 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:11.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:11.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:11.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:11.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:07:11.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:11.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:11.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:07:11.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:11.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:07:11.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:11.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:11.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:11.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:11.621 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:11.621 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:11.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:11.621 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:11.622 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:11.622 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:11.622 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:11.622 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:11.622 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:11.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:11.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:11.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:11.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:07:11.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:11.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:11.736 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:11.736 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:11.736 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:11.736 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:11.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:11.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-09-16T17:07:11.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 1 2024-09-16T17:07:11.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:11.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:11.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:11.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:11.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:11.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:11.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-10 2024-09-16T17:07:11.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:11.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:11.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:11.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:07:11.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:11.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-10 2024-09-16T17:07:11.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:11.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-09-16T17:07:11.856 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-09-16T17:07:11.856 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:12.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:12.312 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:12.312 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:12.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:12.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:12.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:12.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:07:12.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:12.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:12.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:07:12.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:12.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:07:12.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:12.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:07:12.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:12.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:12.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:12.316 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:07:12.316 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:12.316 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:12.317 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:12.317 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:12.317 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:12.317 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:12.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:12.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:12.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:12.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:12.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:07:12.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:12.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:12.440 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:12.440 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:12.440 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:12.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:12.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:12.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-warn-10 2024-09-16T17:07:12.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:07:12.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:12.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:12.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:07:12.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:07:12.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:12.563 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:12.564 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:12.564 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:12.564 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:12.564 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:12.564 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:12.565 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:12.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:07:12.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:12.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-09-16T17:07:12.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:12.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:07:12.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-09-16T17:07:12.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:07:12.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:07:12.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:07:12.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:12.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:12.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:12.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-11 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-11 2024-09-16T17:07:12.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:12.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-09-16T17:07:12.731 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-09-16T17:07:12.731 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:13.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:13.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T17:07:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:13.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-16T17:07:13.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:13.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T17:07:13.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:13.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:13.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:13.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:13.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:13.150 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:13.150 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:13.150 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:13.151 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:13.151 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:13.151 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:13.151 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:13.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:13.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:13.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:13.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:13.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T17:07:13.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:13.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:13.279 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:13.279 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:13.280 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:13.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:13.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:13.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-09-16T17:07:13.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 1 2024-09-16T17:07:13.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:07:13.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-rep 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-warn-11 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-rep obj-warn-11 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-16T17:07:13.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-09-16T17:07:13.412 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-09-16T17:07:13.412 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:07:13.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T17:07:13.839 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:13.839 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:07:13.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T17:07:13.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('3' '1' '0') 2024-09-16T17:07:13.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:07:13.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:07:13.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:07:13.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:07:13.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:07:13.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:07:13.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:07:13.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:13.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:07:13.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:07:13.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:07:13.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:13.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:07:13.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:13.842 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:13.842 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:13.842 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:13.842 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:13.843 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:13.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:07:13.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:13.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:07:13.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:07:13.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:07:13.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:07:13.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:07:13.972 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:13.973 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:13.973 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:13.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:07:13.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:07:13.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr pool-rep obj-warn-11 2024-09-16T17:07:14.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-16T17:07:14.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:07:14.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T17:07:14.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-16T17:07:14.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-16T17:07:14.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:07:14.093 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T17:07:14.093 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T17:07:14.094 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T17:07:14.094 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:14.094 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:14.094 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:14.095 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.3.asok 2024-09-16T17:07:14.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-16T17:07:14.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:07:14.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-09-16T17:07:14.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:14.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-16T17:07:14.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-09-16T17:07:14.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-16T17:07:14.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-16T17:07:14.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-16T17:07:14.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:14.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:14.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-16T17:07:14.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:231: TEST_rados_repair_warning: wait_for_clean 2024-09-16T17:07:14.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:07:14.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:07:14.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:07:14.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:07:14.265 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:07:14.265 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:07:14.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:07:14.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:07:14.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:07:14.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:07:14.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:07:14.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:07:14.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:07:14.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:07:14.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:07:14.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:07:14.771 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:14.771 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:07:14.772 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:07:14.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:07:14.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:07:14.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:07:14.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803828 2024-09-16T17:07:14.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803828 2024-09-16T17:07:14.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803828' 2024-09-16T17:07:14.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:07:14.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:07:15.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607593 2024-09-16T17:07:15.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607593 2024-09-16T17:07:15.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803828 1-51539607593' 2024-09-16T17:07:15.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:07:15.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:07:15.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378664 2024-09-16T17:07:15.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378664 2024-09-16T17:07:15.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803828 1-51539607593 2-81604378664' 2024-09-16T17:07:15.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:07:15.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:07:15.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986204 2024-09-16T17:07:15.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986204 2024-09-16T17:07:15.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803828 1-51539607593 2-81604378664 3-133143986204' 2024-09-16T17:07:15.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:07:15.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803828 2024-09-16T17:07:15.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:07:15.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:07:15.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803828 2024-09-16T17:07:15.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:07:15.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803828 2024-09-16T17:07:15.413 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803828 2024-09-16T17:07:15.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803828' 2024-09-16T17:07:15.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:07:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803826 -lt 25769803828 2024-09-16T17:07:15.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:07:16.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:07:16.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:07:17.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803829 -lt 25769803828 2024-09-16T17:07:17.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:07:17.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:07:17.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607593 2024-09-16T17:07:17.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:07:17.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607593 2024-09-16T17:07:17.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:07:17.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607593 2024-09-16T17:07:17.252 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607593 2024-09-16T17:07:17.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607593' 2024-09-16T17:07:17.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:07:17.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607593 -lt 51539607593 2024-09-16T17:07:17.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:07:17.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378664 2024-09-16T17:07:17.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:07:17.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:07:17.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378664 2024-09-16T17:07:17.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:07:17.693 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378664 2024-09-16T17:07:17.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378664 2024-09-16T17:07:17.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378664' 2024-09-16T17:07:17.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:07:18.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378665 -lt 81604378664 2024-09-16T17:07:18.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:07:18.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-133143986204 2024-09-16T17:07:18.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:07:18.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:07:18.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-133143986204 2024-09-16T17:07:18.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:07:18.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986204 2024-09-16T17:07:18.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 133143986204' 2024-09-16T17:07:18.154 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 133143986204 2024-09-16T17:07:18.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:07:18.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986204 -lt 133143986204 2024-09-16T17:07:18.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:07:18.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:07:18.580 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:07:19.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:07:19.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:07:19.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:07:19.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:07:19.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:07:19.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:07:19.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:07:19.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:07:19.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-16T17:07:19.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:07:19.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:07:19.585 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:07:20.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-16T17:07:20.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:07:20.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:07:20.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: ceph pg 2.0 query 2024-09-16T17:07:20.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-09-16T17:07:20.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: COUNT=22 2024-09-16T17:07:20.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: expr 11 '*' 2 2024-09-16T17:07:20.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: test 22 = 22 2024-09-16T17:07:20.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:234: TEST_rados_repair_warning: flush_pg_stats 2024-09-16T17:07:20.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:07:20.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:07:20.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:07:20.685 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:07:20.685 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:07:20.685 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-16T17:07:20.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:07:20.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:07:20.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:07:20.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803831 2024-09-16T17:07:20.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803831 2024-09-16T17:07:20.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803831' 2024-09-16T17:07:20.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:07:20.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:07:21.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607596 2024-09-16T17:07:21.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607596 2024-09-16T17:07:21.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803831 1-51539607596' 2024-09-16T17:07:21.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:07:21.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:07:21.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378668 2024-09-16T17:07:21.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378668 2024-09-16T17:07:21.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803831 1-51539607596 2-81604378668' 2024-09-16T17:07:21.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:07:21.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:07:21.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986207 2024-09-16T17:07:21.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986207 2024-09-16T17:07:21.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803831 1-51539607596 2-81604378668 3-133143986207' 2024-09-16T17:07:21.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:07:21.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803831 2024-09-16T17:07:21.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:07:21.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:07:21.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803831 2024-09-16T17:07:21.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:07:21.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803831 2024-09-16T17:07:21.360 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803831 2024-09-16T17:07:21.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-09-16T17:07:21.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:07:21.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803829 -lt 25769803831 2024-09-16T17:07:21.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:07:22.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:07:22.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:07:23.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803832 -lt 25769803831 2024-09-16T17:07:23.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:07:23.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607596 2024-09-16T17:07:23.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:07:23.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:07:23.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607596 2024-09-16T17:07:23.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:07:23.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607596 2024-09-16T17:07:23.186 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607596 2024-09-16T17:07:23.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607596' 2024-09-16T17:07:23.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:07:23.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607597 -lt 51539607596 2024-09-16T17:07:23.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:07:23.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378668 2024-09-16T17:07:23.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:07:23.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:07:23.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378668 2024-09-16T17:07:23.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:07:23.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378668 2024-09-16T17:07:23.592 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378668 2024-09-16T17:07:23.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378668' 2024-09-16T17:07:23.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:07:24.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378668 -lt 81604378668 2024-09-16T17:07:24.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:07:24.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-133143986207 2024-09-16T17:07:24.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:07:24.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:07:24.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-133143986207 2024-09-16T17:07:24.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:07:24.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986207 2024-09-16T17:07:24.018 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 133143986207 2024-09-16T17:07:24.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 133143986207' 2024-09-16T17:07:24.019 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:07:24.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986207 -lt 133143986207 2024-09-16T17:07:24.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-09-16T17:07:24.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-16T17:07:24.857 INFO:tasks.workunit.client.0.smithi039.stderr:dumped all 2024-09-16T17:07:24.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: COUNT=33 2024-09-16T17:07:24.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: expr 11 '*' 3 2024-09-16T17:07:24.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: test 33 = 33 2024-09-16T17:07:24.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:240: TEST_rados_repair_warning: CHECKTIME=10 2024-09-16T17:07:24.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:241: TEST_rados_repair_warning: LOOPS=0 2024-09-16T17:07:24.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:242: TEST_rados_repair_warning: true 2024-09-16T17:07:24.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:244: TEST_rados_repair_warning: sleep 1 2024-09-16T17:07:25.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: ceph health 2024-09-16T17:07:25.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 2 OSDs' 2024-09-16T17:07:26.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:247: TEST_rados_repair_warning: break 2024-09-16T17:07:26.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: ceph health detail 2024-09-16T17:07:26.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: expr 11 '*' 2 2024-09-16T17:07:26.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: grep -q 'osd.3 had 22 reads repaired' 2024-09-16T17:07:26.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: grep -q 'osd.1 had 11 reads repaired' 2024-09-16T17:07:26.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: ceph health detail 2024-09-16T17:07:27.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:259: TEST_rados_repair_warning: delete_pool pool-rep 2024-09-16T17:07:27.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-09-16T17:07:27.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-09-16T17:07:28.049 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'pool-rep' does not exist 2024-09-16T17:07:28.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-16T17:07:28.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:28.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:07:28.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-16T17:07:28.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:07:28.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:07:28.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:07:28.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:07:28.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:07:28.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:07:28.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:07:28.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:07:28.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:07:28.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:07:28.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:07:28.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:07:28.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:07:28.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:07:28.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:07:28.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:07:28.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:07:28.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:07:28.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-16T17:07:28.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:07:28.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:28.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:28.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383662 2024-09-16T17:07:28.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:07:28.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:07:28.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-16T17:07:28.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-16T17:07:28.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:28.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-16T17:07:28.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:28.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:07:28.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-16T17:07:28.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:07:28.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:07:28.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:07:28.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:07:28.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:07:28.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:07:28.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:07:28.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:07:28.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:07:28.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:07:28.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:07:28.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:07:28.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:07:28.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:07:28.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:07:28.218 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:07:28.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:07:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:07:28.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-16T17:07:28.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:07:28.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:28.221 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:28.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383662 2024-09-16T17:07:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:07:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:07:28.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-16T17:07:28.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:07:28.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:28.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:28.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.383662 2024-09-16T17:07:28.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:07:28.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T17:07:28.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:07:28.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-16T17:07:28.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-16T17:07:28.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:28.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:07:28.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:07:28.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:07:28.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-16T17:07:28.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-16T17:07:28.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:07:28.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:07:28.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:07:28.599 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:07:28.599 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:28.599 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:28.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:07:28.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-16T17:07:28.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:07:28.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:07:28.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:07:28.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:07:28.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:07:28.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:07:28.804 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:07:28.804 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:07:28.804 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:07:28.804 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:28.804 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:28.805 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:28.805 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-mon.a.asok 2024-09-16T17:07:28.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:07:28.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383662/ceph-mon.a.asok config get fsid 2024-09-16T17:07:28.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:07:28.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:07:28.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:07:28.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:07:28.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:07:28.925 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:07:28.925 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:07:28.925 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:07:28.926 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:07:28.926 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:28.926 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:28.926 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-mon.a.asok 2024-09-16T17:07:28.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:07:28.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383662/ceph-mon.a.asok config get mon_host 2024-09-16T17:07:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-16T17:07:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:07:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:07:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:07:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-16T17:07:29.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:07:29.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:07:29.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:07:29.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:07:29.951 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:07:29.951 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:29.951 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:29.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:07:29.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:07:29.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:07:29.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-16T17:07:30.286 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T17:07:30.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_backfill_unfound td/osd-rep-recov-eio.sh 2024-09-16T17:07:30.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:264: TEST_rep_backfill_unfound: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:30.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:265: TEST_rep_backfill_unfound: local objname=myobject 2024-09-16T17:07:30.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:266: TEST_rep_backfill_unfound: local lastobj=300 2024-09-16T17:07:30.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:268: TEST_rep_backfill_unfound: local testobj=obj250 2024-09-16T17:07:30.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:270: TEST_rep_backfill_unfound: export CEPH_ARGS 2024-09-16T17:07:30.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:271: TEST_rep_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T17:07:30.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:272: TEST_rep_backfill_unfound: setup_osds 3 2024-09-16T17:07:30.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-09-16T17:07:30.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-16T17:07:30.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-16T17:07:30.304 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-09-16T17:07:30.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-09-16T17:07:30.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T17:07:30.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-16T17:07:30.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:07:30.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:07:30.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:07:30.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:07:30.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:07:30.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:07:30.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-16T17:07:30.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:07:30.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b395b92d-1ebb-48fe-a0fa-20cda55dae9c 2024-09-16T17:07:30.312 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 b395b92d-1ebb-48fe-a0fa-20cda55dae9c 2024-09-16T17:07:30.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b395b92d-1ebb-48fe-a0fa-20cda55dae9c' 2024-09-16T17:07:30.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:07:30.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSZehmJTsHFBAA0L2mYC8gLcrTrHTmPEPKyQ== 2024-09-16T17:07:30.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSZehmJTsHFBAA0L2mYC8gLcrTrHTmPEPKyQ=="}' 2024-09-16T17:07:30.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b395b92d-1ebb-48fe-a0fa-20cda55dae9c -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-16T17:07:30.727 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:07:30.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-16T17:07:30.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSZehmJTsHFBAA0L2mYC8gLcrTrHTmPEPKyQ== --osd-uuid b395b92d-1ebb-48fe-a0fa-20cda55dae9c 2024-09-16T17:07:30.781 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:30.778+0000 7efd0fe077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:07:30.783 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:30.782+0000 7efd0fe077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:07:30.784 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:30.782+0000 7efd0fe077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:07:30.785 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:30.782+0000 7efd0fe077c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T17:07:30.785 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:30.782+0000 7efd0fe077c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T17:07:30.785 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:30.782+0000 7efd0fe077c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T17:07:30.875 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:30.874+0000 7efd0fe077c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-16T17:07:36.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-16T17:07:36.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:07:36.701 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:07:36.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:07:36.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:07:37.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:07:37.258 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:07:37.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:07:37.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:07:37.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:07:37.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:07:37.292 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:37.290+0000 7ff679cfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:07:37.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:37.314+0000 7ff679cfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:07:37.318 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:37.314+0000 7ff679cfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:07:37.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:07:37.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:07:37.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:07:37.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:07:37.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:07:37.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:37.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:07:37.733 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:07:37.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:37.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:37.900 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:37.898+0000 7ff679cfb7c0 -1 Falling back to public interface 2024-09-16T17:07:38.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:39.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:39.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:39.177 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:07:39.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:07:39.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:39.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:39.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:40.611 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:07:40.610+0000 7ff679cfb7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:07:40.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:40.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:40.613 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:07:40.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:07:40.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:40.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:41.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:42.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:42.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:42.221 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:07:42.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:07:42.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:42.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:42.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:43.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:43.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:43.673 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:07:43.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:07:43.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:43.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:44.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:45.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:45.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:45.116 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:07:45.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:07:45.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:45.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:45.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:46.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:46.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:46.676 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:07:46.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:07:46.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:46.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:47.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:48.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:48.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:48.170 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:07:48.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:07:48.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:48.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:48.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:49.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:49.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:49.606 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:07:49.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:07:49.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:49.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:50.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:51.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:51.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:51.072 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:07:51.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:07:51.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:51.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:51.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:52.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:52.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:52.678 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:07:52.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:07:52.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:52.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:53.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:54.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:54.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:54.115 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:07:54.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:07:54.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:54.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:54.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:55.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:55.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:55.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:07:55.936 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:07:55.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:55.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:56.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:57.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:57.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:57.635 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:07:57.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:07:57.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:57.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:07:58.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:07:59.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:07:59.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:07:59.807 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:07:59.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:07:59.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:07:59.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:08:00.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:01.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:01.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:01.679 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:08:01.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:08:01.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:01.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:08:02.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:03.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:03.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:03.730 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:03.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:08:04.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:05.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:05.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:05.394 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:08:05.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:08:05.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:05.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:08:05.955 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/69372349,v1:127.0.0.1:6803/69372349] [v2:127.0.0.1:6804/69372349,v1:127.0.0.1:6805/69372349] exists,up b395b92d-1ebb-48fe-a0fa-20cda55dae9c 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:08:05.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:08:05.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-16T17:08:05.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-16T17:08:05.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:08:05.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:08:05.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:08:05.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:08:05.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:08:05.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:08:05.959 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:08:05.959 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:08:05.959 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:08:05.959 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:08:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-16T17:08:05.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:08:05.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b772a9a6-d399-4d08-ad08-0b3bbc16962c 2024-09-16T17:08:05.961 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 b772a9a6-d399-4d08-ad08-0b3bbc16962c 2024-09-16T17:08:05.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b772a9a6-d399-4d08-ad08-0b3bbc16962c' 2024-09-16T17:08:05.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:08:05.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1Zehmbq6mOhAAtXO17709x2BrptWd+J0giA== 2024-09-16T17:08:05.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1Zehmbq6mOhAAtXO17709x2BrptWd+J0giA=="}' 2024-09-16T17:08:05.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b772a9a6-d399-4d08-ad08-0b3bbc16962c -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-16T17:08:06.533 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:08:06.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-16T17:08:06.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1Zehmbq6mOhAAtXO17709x2BrptWd+J0giA== --osd-uuid b772a9a6-d399-4d08-ad08-0b3bbc16962c 2024-09-16T17:08:06.588 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:06.586+0000 7f4d5c0217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:06.614 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:06.610+0000 7f4d5c0217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:06.616 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:06.614+0000 7f4d5c0217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:06.616 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:06.614+0000 7f4d5c0217c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T17:08:06.616 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:06.614+0000 7f4d5c0217c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T17:08:06.616 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:06.614+0000 7f4d5c0217c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T17:08:06.706 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:06.702+0000 7f4d5c0217c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-16T17:08:13.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-16T17:08:13.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:08:13.140 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T17:08:13.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T17:08:13.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:08:13.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T17:08:13.724 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:08:13.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:08:13.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:08:13.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:08:13.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:08:13.759 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:13.758+0000 7efe001877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:13.761 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:13.758+0000 7efe001877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:13.763 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:13.762+0000 7efe001877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:14.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T17:08:14.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:08:14.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:08:14.188 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:08:14.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:08:14.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:08:14.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:14.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:08:14.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:14.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:14.328 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:14.325+0000 7efe001877c0 -1 Falling back to public interface 2024-09-16T17:08:14.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:15.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:15.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:15.687 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:08:15.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:08:15.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:15.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:16.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:16.933 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:16.929+0000 7efe001877c0 -1 osd.1 0 log_to_monitors true 2024-09-16T17:08:17.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:17.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:17.103 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:08:17.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:08:17.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:17.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:17.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:18.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:18.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:18.823 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:08:18.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:08:18.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:18.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:19.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:20.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:20.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:20.372 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:08:20.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:08:20.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:20.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:21.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:22.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:22.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:22.006 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:08:22.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:08:22.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:22.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:22.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:23.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:23.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:23.442 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:08:23.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:08:23.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:23.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:24.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:25.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:25.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:25.008 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:08:25.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:08:25.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:25.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:25.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:26.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:26.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:26.557 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:08:26.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:08:26.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:26.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:27.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:28.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:28.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:28.194 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:08:28.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:08:28.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:28.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:28.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:29.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:29.643 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:08:29.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:29.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:08:29.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:29.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:30.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:31.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:31.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:31.085 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:08:31.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:08:31.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:31.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:31.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:32.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:32.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:32.976 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:08:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:08:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:33.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:34.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:34.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:34.832 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:08:34.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:08:34.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:34.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:36.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:37.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:37.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:37.332 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:08:37.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:08:37.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:37.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:38.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:39.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:39.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:39.078 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:08:39.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:08:39.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:39.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:40.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:41.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:41.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:41.071 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:08:41.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:08:41.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:41.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:41.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:42.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:42.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:42.926 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:08:42.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:08:42.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:42.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:08:43.608 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/623598424,v1:127.0.0.1:6811/623598424] [v2:127.0.0.1:6812/623598424,v1:127.0.0.1:6813/623598424] exists,up b772a9a6-d399-4d08-ad08-0b3bbc16962c 2024-09-16T17:08:43.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:08:43.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:08:43.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-16T17:08:43.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:08:43.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:08:43.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:08:43.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:08:43.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:08:43.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:08:43.612 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:08:43.612 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:08:43.612 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:08:43.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:08:43.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:08:43.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:08:43.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:08:43.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:08:43.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:08:43.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:08:43.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:08:43.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:08:43.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:08:43.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:08:43.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:08:43.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:08:43.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-16T17:08:43.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:08:43.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=af3c4fb7-dfc0-4c55-91bb-bf9ac8b395db 2024-09-16T17:08:43.615 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 af3c4fb7-dfc0-4c55-91bb-bf9ac8b395db 2024-09-16T17:08:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 af3c4fb7-dfc0-4c55-91bb-bf9ac8b395db' 2024-09-16T17:08:43.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:08:43.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbZuhmKAcUJhAA3GJfpfAH819FuEvzDtS2BA== 2024-09-16T17:08:43.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbZuhmKAcUJhAA3GJfpfAH819FuEvzDtS2BA=="}' 2024-09-16T17:08:43.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new af3c4fb7-dfc0-4c55-91bb-bf9ac8b395db -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-16T17:08:44.250 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:08:44.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-16T17:08:44.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbZuhmKAcUJhAA3GJfpfAH819FuEvzDtS2BA== --osd-uuid af3c4fb7-dfc0-4c55-91bb-bf9ac8b395db 2024-09-16T17:08:44.298 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:44.293+0000 7f530ea497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:44.300 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:44.297+0000 7f530ea497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:44.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:44.297+0000 7f530ea497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:44.301 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:44.297+0000 7f530ea497c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T17:08:44.302 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:44.297+0000 7f530ea497c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T17:08:44.302 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:44.297+0000 7f530ea497c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T17:08:44.399 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:44.397+0000 7f530ea497c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-16T17:08:51.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-16T17:08:51.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:08:51.001 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T17:08:51.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T17:08:51.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:08:51.634 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T17:08:51.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T17:08:51.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:08:51.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:08:51.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:08:51.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:08:51.671 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:51.669+0000 7fb9e877b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:51.705 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:51.701+0000 7fb9e877b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:51.707 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:51.705+0000 7fb9e877b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:08:52.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T17:08:52.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:08:52.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:08:52.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:08:52.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:08:52.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:52.103 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:08:52.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:08:52.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:52.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:08:52.300 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:52.297+0000 7fb9e877b7c0 -1 Falling back to public interface 2024-09-16T17:08:52.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:53.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:53.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:53.598 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:08:53.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:08:53.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:53.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:08:54.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:55.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:55.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:55.035 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:08:55.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:08:55.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:55.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:08:55.090 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:08:55.069+0000 7fb9e877b7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T17:08:55.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:56.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:56.704 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:08:56.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:08:56.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:56.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:08:57.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:08:58.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:08:58.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:08:58.475 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:08:58.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:08:58.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:08:58.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:08:59.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:00.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:00.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:00.144 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:09:00.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:09:00.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:00.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:00.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:01.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:01.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:01.609 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:09:01.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:09:01.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:01.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:02.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:03.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:03.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:03.155 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:09:03.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:09:03.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:03.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:03.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:04.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:04.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:04.651 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:09:04.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:09:04.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:04.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:05.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:06.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:06.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:06.261 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:09:06.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:09:06.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:06.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:06.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:07.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:07.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:07.730 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:09:07.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:09:07.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:07.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:08.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:09.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:09.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:09.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:09:09.179 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:09:09.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:09.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:09.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:10.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:09:10.655 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:09:10.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:10.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:11.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:12.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:12.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:12.706 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:09:12.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:09:12.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:12.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:14.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:15.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:15.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:15.124 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:09:15.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:09:15.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:15.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:15.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:16.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:16.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:16.964 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:09:16.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:09:16.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:16.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:17.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:18.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:18.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:18.957 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:09:18.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:09:18.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:18.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:19.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:20.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:20.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:20.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:09:20.833 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:09:20.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:20.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:21.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:09:22.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:09:22.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:09:22.382 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:09:22.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T17:09:22.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:09:22.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:09:22.943 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4289498475,v1:127.0.0.1:6819/4289498475] [v2:127.0.0.1:6820/4289498475,v1:127.0.0.1:6821/4289498475] exists,up af3c4fb7-dfc0-4c55-91bb-bf9ac8b395db 2024-09-16T17:09:22.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:09:22.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:09:22.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:09:22.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-16T17:09:22.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:09:22.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:09:22.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:09:22.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:09:22.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:09:22.946 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:09:22.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:09:22.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:09:22.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:09:23.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:09:23.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:09:23.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:09:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:09:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:09:23.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:09:23.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:09:23.529 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:09:23.529 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:09:23.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:09:23.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:09:23.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:09:23.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803796 2024-09-16T17:09:23.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803796 2024-09-16T17:09:23.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803796' 2024-09-16T17:09:23.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:09:23.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:09:23.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607570 2024-09-16T17:09:23.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607570 2024-09-16T17:09:23.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803796 1-51539607570' 2024-09-16T17:09:23.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:09:23.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:09:24.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378633 2024-09-16T17:09:24.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378633 2024-09-16T17:09:24.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803796 1-51539607570 2-81604378633' 2024-09-16T17:09:24.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:09:24.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803796 2024-09-16T17:09:24.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:09:24.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:09:24.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803796 2024-09-16T17:09:24.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:09:24.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803796 2024-09-16T17:09:24.090 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803796 2024-09-16T17:09:24.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803796' 2024-09-16T17:09:24.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:09:24.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803794 -lt 25769803796 2024-09-16T17:09:24.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:09:25.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:09:25.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:09:25.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803796 -lt 25769803796 2024-09-16T17:09:25.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:09:26.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607570 2024-09-16T17:09:26.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:09:26.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:09:26.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607570 2024-09-16T17:09:26.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:09:26.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607570 2024-09-16T17:09:26.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607570' 2024-09-16T17:09:26.004 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607570 2024-09-16T17:09:26.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:09:26.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607570 -lt 51539607570 2024-09-16T17:09:26.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:09:26.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378633 2024-09-16T17:09:26.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:09:26.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:09:26.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378633 2024-09-16T17:09:26.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:09:26.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378633 2024-09-16T17:09:26.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378633' 2024-09-16T17:09:26.429 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378633 2024-09-16T17:09:26.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:09:26.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378634 -lt 81604378633 2024-09-16T17:09:26.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:09:26.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:09:26.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:09:27.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T17:09:27.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:09:27.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:09:27.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:09:27.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:09:27.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:09:27.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:09:27.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:09:27.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:09:27.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:09:27.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:09:27.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:09:28.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T17:09:28.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:09:28.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:09:28.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:274: TEST_rep_backfill_unfound: local poolname=test-pool 2024-09-16T17:09:28.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:275: TEST_rep_backfill_unfound: create_pool test-pool 1 1 2024-09-16T17:09:28.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-09-16T17:09:29.157 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test-pool' already exists 2024-09-16T17:09:29.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T17:09:30.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:276: TEST_rep_backfill_unfound: wait_for_clean 2024-09-16T17:09:30.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:09:30.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:09:30.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:09:30.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:09:30.179 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:09:30.179 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:09:30.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:09:30.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:09:30.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:09:30.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:09:30.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:09:30.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:09:30.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:09:30.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:09:30.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:09:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:09:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:09:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:09:30.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:09:30.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:09:30.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:09:30.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803800 2024-09-16T17:09:30.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803800 2024-09-16T17:09:30.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803800' 2024-09-16T17:09:30.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:09:30.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:09:31.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607573 2024-09-16T17:09:31.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607573 2024-09-16T17:09:31.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803800 1-51539607573' 2024-09-16T17:09:31.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:09:31.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:09:31.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378636 2024-09-16T17:09:31.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378636 2024-09-16T17:09:31.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803800 1-51539607573 2-81604378636' 2024-09-16T17:09:31.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:09:31.344 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803800 2024-09-16T17:09:31.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:09:31.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:09:31.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803800 2024-09-16T17:09:31.346 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:09:31.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803800 2024-09-16T17:09:31.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803800' 2024-09-16T17:09:31.348 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803800 2024-09-16T17:09:31.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:09:31.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803798 -lt 25769803800 2024-09-16T17:09:31.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:09:32.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:09:32.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:09:33.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803800 -lt 25769803800 2024-09-16T17:09:33.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:09:33.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607573 2024-09-16T17:09:33.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:09:33.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:09:33.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607573 2024-09-16T17:09:33.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:09:33.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607573 2024-09-16T17:09:33.328 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607573 2024-09-16T17:09:33.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607573' 2024-09-16T17:09:33.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:09:33.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607573 -lt 51539607573 2024-09-16T17:09:33.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:09:33.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378636 2024-09-16T17:09:33.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:09:33.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:09:33.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378636 2024-09-16T17:09:33.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:09:33.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378636 2024-09-16T17:09:33.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378636' 2024-09-16T17:09:33.769 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378636 2024-09-16T17:09:33.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:09:34.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378637 -lt 81604378636 2024-09-16T17:09:34.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:09:34.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:09:34.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:09:34.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:09:34.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:09:34.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:09:34.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:09:34.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:09:34.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:09:34.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:09:34.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:09:35.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-16T17:09:35.164 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:09:35.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:09:35.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:09:35.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-16T17:09:35.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:09:35.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:09:35.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:278: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-09-16T17:09:36.123 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T17:09:36.123 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:09:22.467775+0000 0'0 24:28 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-18T02:50:38.073069+0000 0 0 2024-09-16T17:09:36.123 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:09:22.468302+0000 0'0 24:28 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:36:11.336334+0000 0 0 2024-09-16T17:09:36.123 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:09:22.468252+0000 0'0 24:15 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-18T02:33:26.605539+0000 0 0 2024-09-16T17:09:36.124 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:09:22.424411+0000 0'0 24:28 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-18T00:44:41.717753+0000 0 0 2024-09-16T17:09:36.124 INFO:tasks.workunit.client.0.smithi039.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:09:30.214234+0000 0'0 25:11 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-16T17:09:28.850562+0000 0'0 2024-09-16T17:09:28.850562+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:24:28.980531+0000 0 0 2024-09-16T17:09:36.124 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:09:22.467632+0000 0'0 24:28 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:15:05.482315+0000 0 0 2024-09-16T17:09:36.124 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:09:23.555095+0000 0'0 23:36 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:07:30.673650+0000 0 0 2024-09-16T17:09:36.124 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:09:22.467101+0000 0'0 24:15 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:14:09.871805+0000 0 0 2024-09-16T17:09:36.124 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:09:22.468819+0000 0'0 24:28 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-18T02:18:23.990659+0000 0 0 2024-09-16T17:09:36.124 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T17:09:36.124 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T17:09:36.124 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T17:09:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:280: TEST_rep_backfill_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-09-16T17:09:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:09:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-09-16T17:09:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-09-16T17:09:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:09:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:09:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:09:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:09:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:09:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:09:36.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:09:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:09:36.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:36.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: get_osds test-pool myobject 2024-09-16T17:09:36.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-16T17:09:36.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T17:09:36.324 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:09:36.324 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-09-16T17:09:36.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-16T17:09:36.800 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:09:36.800 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:09:36.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-09-16T17:09:36.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: initial_osds=('2' '1' '0') 2024-09-16T17:09:36.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: local -a initial_osds 2024-09-16T17:09:36.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:283: TEST_rep_backfill_unfound: local last_osd=0 2024-09-16T17:09:36.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:284: TEST_rep_backfill_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-09-16T17:09:36.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:09:36.801 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:09:36.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:09:36.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:09:36.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:09:37.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:09:37.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:285: TEST_rep_backfill_unfound: ceph osd down 0 2024-09-16T17:09:37.884 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T17:09:37.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:286: TEST_rep_backfill_unfound: ceph osd out 0 2024-09-16T17:09:38.624 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T17:09:38.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:288: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-09-16T17:09:39.146 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T17:09:39.146 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:09:37.944747+0000 0'0 28:34 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T19:16:37.987012+0000 0 0 2024-09-16T17:09:39.147 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:09:37.944720+0000 0'0 28:34 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-18T01:04:27.903836+0000 0 0 2024-09-16T17:09:39.147 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:09:22.468252+0000 0'0 24:15 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-18T02:33:26.605539+0000 0 0 2024-09-16T17:09:39.147 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:09:37.944680+0000 0'0 28:34 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:53:11.763149+0000 0 0 2024-09-16T17:09:39.147 INFO:tasks.workunit.client.0.smithi039.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 active+clean 2024-09-16T17:09:30.214234+0000 25'1 25:12 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-16T17:09:28.850562+0000 0'0 2024-09-16T17:09:28.850562+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:24:28.980531+0000 0 0 2024-09-16T17:09:39.147 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:09:37.944374+0000 0'0 28:34 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:37:43.040345+0000 0 0 2024-09-16T17:09:39.147 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:09:37.944378+0000 0'0 29:31 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:24:51.592232+0000 0 0 2024-09-16T17:09:39.147 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:09:22.467101+0000 0'0 24:15 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:14:09.871805+0000 0 0 2024-09-16T17:09:39.147 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:09:37.944388+0000 0'0 28:34 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:49:23.337413+0000 0 0 2024-09-16T17:09:39.147 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T17:09:39.147 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T17:09:39.147 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T17:09:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:290: TEST_rep_backfill_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-09-16T17:09:39.165 INFO:tasks.workunit.client.0.smithi039.stderr:4+0 records in 2024-09-16T17:09:39.165 INFO:tasks.workunit.client.0.smithi039.stderr:4+0 records out 2024-09-16T17:09:39.165 INFO:tasks.workunit.client.0.smithi039.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000301035 s, 13.6 MB/s 2024-09-16T17:09:39.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: seq 1 300 2024-09-16T17:09:39.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:39.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:39.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:39.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:39.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:39.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:39.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:39.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:39.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:39.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:40.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:40.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:40.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:40.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:40.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:40.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:40.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:40.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:40.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:40.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:40.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:40.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:41.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:41.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:41.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:41.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:41.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:41.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:41.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:41.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:41.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:41.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:41.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:41.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:42.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:42.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:42.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:42.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:42.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:42.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:42.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:42.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:42.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:42.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:42.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:42.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:43.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:43.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:43.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:43.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:43.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:43.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:43.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:43.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:43.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:43.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:43.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:43.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:44.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:44.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:44.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:44.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:44.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:44.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:44.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:44.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:44.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:44.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:44.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:44.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:45.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:45.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:45.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:45.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:45.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:45.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:45.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:45.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:45.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:45.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:45.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:45.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:46.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:46.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:46.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:46.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:46.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:46.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:46.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:46.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:46.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:46.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:46.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:46.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:47.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:47.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:47.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:47.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:47.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:47.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:47.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:47.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:47.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:47.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:48.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:48.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:48.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:48.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:48.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:48.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:48.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:48.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:48.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:48.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:48.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:48.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:48.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:48.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:49.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:49.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:49.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:49.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:49.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:49.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:49.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:49.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:49.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:49.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:50.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:50.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:50.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:50.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:50.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:50.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:50.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:50.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:50.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:50.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:50.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:50.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:50.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:51.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:51.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:51.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:51.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:51.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:51.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:51.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:51.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:51.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:51.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:51.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:51.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:52.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:52.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:52.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:52.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:52.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:52.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:52.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:52.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:52.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:52.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:52.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:52.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:52.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:53.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:53.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:53.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:53.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:53.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:53.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:53.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:53.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:53.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:53.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:53.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:53.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:54.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:54.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:54.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:54.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:54.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:54.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:54.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:54.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:54.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:54.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:54.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:54.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:55.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:55.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:55.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:55.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:55.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:55.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:55.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:55.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj101 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:55.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:55.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj102 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:56.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:56.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj103 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:56.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:56.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj104 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:56.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:56.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj105 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:56.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:56.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj106 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:56.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:56.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj107 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:56.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:56.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj108 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:56.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:56.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj109 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:57.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:57.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj110 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:57.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:57.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj111 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:57.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:57.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj112 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:57.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:57.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj113 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:57.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:57.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj114 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:57.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:57.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj115 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:58.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:58.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj116 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:58.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:58.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj117 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:58.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:58.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj118 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:58.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:58.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj119 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:58.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:58.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj120 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:58.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:58.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj121 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:59.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:59.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj122 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:59.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:59.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj123 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:59.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:59.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj124 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:59.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:59.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj125 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:59.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:59.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj126 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:09:59.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:09:59.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj127 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:00.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:00.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj128 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:00.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:00.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj129 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:00.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:00.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj130 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:00.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:00.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj131 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:00.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:00.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj132 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:00.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:00.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj133 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:00.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:00.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj134 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:01.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:01.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj135 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:01.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:01.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj136 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:01.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:01.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj137 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:01.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:01.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj138 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:01.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:01.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj139 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:01.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:01.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj140 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:02.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:02.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj141 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:02.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:02.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj142 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:02.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:02.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj143 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:02.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:02.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj144 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:02.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:02.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj145 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:02.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:02.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj146 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:03.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:03.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj147 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:03.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:03.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj148 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:03.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:03.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj149 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:03.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:03.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj150 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:03.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:03.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj151 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:03.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:03.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj152 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:04.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:04.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj153 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:04.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:04.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj154 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:04.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:04.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj155 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:04.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:04.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj156 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:04.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:04.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj157 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:04.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:04.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj158 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:05.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:05.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj159 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:05.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:05.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj160 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:05.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:05.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj161 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:05.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:05.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj162 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:05.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:05.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj163 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:05.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:05.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj164 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:05.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:05.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj165 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:06.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:06.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj166 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:06.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:06.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj167 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:06.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:06.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj168 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:06.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:06.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj169 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:06.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:06.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj170 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:06.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:06.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj171 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:07.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:07.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj172 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:07.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:07.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj173 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:07.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:07.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj174 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:07.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj175 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:07.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:07.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj176 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:07.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:07.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj177 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:08.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:08.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj178 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:08.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:08.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj179 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:08.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:08.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj180 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:08.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:08.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj181 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:08.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:08.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj182 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:08.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:08.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj183 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:09.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:09.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj184 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:09.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:09.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj185 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:09.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:09.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj186 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:09.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:09.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj187 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:09.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:09.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj188 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:09.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj189 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:10.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:10.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj190 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:10.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:10.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj191 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:10.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:10.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj192 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:10.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:10.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj193 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:10.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:10.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj194 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:10.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:10.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj195 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:11.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:11.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj196 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:11.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:11.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj197 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:11.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:11.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj198 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:11.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:11.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj199 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:11.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:11.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj200 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:11.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:11.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj201 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:12.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:12.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj202 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:12.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:12.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj203 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:12.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:12.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj204 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:12.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:12.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj205 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:12.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:12.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj206 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:12.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:12.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj207 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:13.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:13.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj208 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:13.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:13.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj209 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:13.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:13.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj210 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:13.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:13.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj211 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:13.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:13.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj212 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:13.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:13.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj213 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:13.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:13.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj214 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:14.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:14.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj215 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:14.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:14.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj216 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:14.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:14.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj217 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:14.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj218 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:14.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:14.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj219 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:14.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:14.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj220 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:15.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:15.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj221 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:15.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:15.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj222 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:15.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:15.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj223 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:15.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:15.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj224 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:15.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:15.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj225 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:15.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:15.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj226 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:16.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:16.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj227 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:16.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:16.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj228 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:16.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:16.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj229 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:16.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:16.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj230 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:16.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj231 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:16.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:16.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj232 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:17.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:17.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj233 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:17.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:17.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj234 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:17.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:17.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj235 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:17.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:17.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj236 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:17.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:17.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj237 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:18.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:18.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj238 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:18.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:18.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj239 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:18.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:18.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj240 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:18.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:18.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj241 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:18.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:18.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj242 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:18.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:18.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj243 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:19.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:19.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj244 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:19.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:19.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj245 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:19.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:19.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj246 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:19.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:19.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj247 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:19.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:19.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj248 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:20.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:20.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj249 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:20.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:20.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj250 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:20.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:20.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj251 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:20.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:20.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj252 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:20.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:20.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj253 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:20.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:20.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj254 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:21.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:21.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj255 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:21.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:21.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj256 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:21.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:21.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj257 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:21.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:21.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj258 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:21.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:21.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj259 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:21.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:21.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj260 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:22.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:22.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj261 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:22.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:22.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj262 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:22.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:22.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj263 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:22.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:22.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj264 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:22.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:22.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj265 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:22.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:22.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj266 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:23.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:23.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj267 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:23.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:23.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj268 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:23.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:23.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj269 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:23.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:23.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj270 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:23.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:23.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj271 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj272 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:23.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:23.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj273 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:24.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:24.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj274 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:24.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:24.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj275 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:24.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:24.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj276 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:24.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:24.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj277 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:24.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:24.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj278 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:24.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:24.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj279 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:25.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:25.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj280 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:25.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:25.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj281 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:25.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:25.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj282 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:25.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:25.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj283 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:25.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:25.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj284 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:25.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj285 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:25.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:25.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj286 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:26.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:26.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj287 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:26.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:26.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj288 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:26.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:26.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj289 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:26.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:26.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj290 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:26.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:26.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj291 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:26.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:26.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj292 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:27.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:27.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj293 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:27.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:27.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj294 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:27.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:27.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj295 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:27.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:27.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj296 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:27.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:27.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj297 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:27.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:27.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj298 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:28.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:28.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj299 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:28.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:10:28.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj300 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:10:28.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:296: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 0 2024-09-16T17:10:28.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:10:28.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:10:28.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:10:28.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:10:28.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=test-pool 2024-09-16T17:10:28.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:10:28.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj250 2024-09-16T17:10:28.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:10:28.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:10:28.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:10:28.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:10:28.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:10:28.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds test-pool obj250 2024-09-16T17:10:28.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-16T17:10:28.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-16T17:10:28.373 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-09-16T17:10:28.373 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:10:28.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-16T17:10:28.822 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-09-16T17:10:28.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-16T17:10:28.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('2' '1') 2024-09-16T17:10:28.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:10:28.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=2 2024-09-16T17:10:28.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:10:28.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:10:28.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-09-16T17:10:28.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:10:28.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-16T17:10:28.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:10:28.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-09-16T17:10:28.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:10:28.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:10:28.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:10:28.825 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-09-16T17:10:28.825 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T17:10:28.825 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T17:10:28.825 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:10:28.826 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:10:28.826 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:10:28.826 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.2.asok 2024-09-16T17:10:28.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:10:28.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:10:28.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:10:28.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.2 2024-09-16T17:10:28.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T17:10:28.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T17:10:28.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:10:28.955 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:10:28.955 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:10:28.955 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:10:28.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.2.asok 2024-09-16T17:10:28.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:10:28.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.2.asok injectdataerr test-pool obj250 2024-09-16T17:10:29.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:297: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 1 2024-09-16T17:10:29.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:10:29.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:10:29.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:10:29.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:10:29.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=test-pool 2024-09-16T17:10:29.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:10:29.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj250 2024-09-16T17:10:29.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:10:29.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:10:29.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:10:29.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:10:29.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:10:29.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds test-pool obj250 2024-09-16T17:10:29.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-16T17:10:29.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-16T17:10:29.077 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-09-16T17:10:29.077 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:10:29.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-16T17:10:29.509 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-09-16T17:10:29.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-16T17:10:29.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('2' '1') 2024-09-16T17:10:29.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:10:29.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:10:29.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:10:29.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:10:29.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:10:29.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:10:29.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:10:29.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:10:29.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:10:29.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:10:29.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:10:29.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:10:29.513 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:10:29.513 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:10:29.513 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:10:29.513 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:10:29.513 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:10:29.513 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:10:29.513 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:10:29.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:10:29.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:10:29.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:10:29.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:10:29.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:10:29.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:10:29.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:10:29.640 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:10:29.641 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:10:29.641 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:10:29.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:10:29.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:10:29.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr test-pool obj250 2024-09-16T17:10:29.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:299: TEST_rep_backfill_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-09-16T17:10:29.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:10:29.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T17:10:29.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T17:10:29.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T17:10:29.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-16T17:10:29.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T17:10:29.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:10:29.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:10:29.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:10:29.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-16T17:10:29.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-16T17:10:29.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T17:10:29.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T17:10:29.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:10:29.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T17:10:29.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:10:29.761 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:10:29.762 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:10:29.762 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:10:29.762 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T17:10:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-16T17:10:29.764 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:10:29.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T17:10:29.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:10:29.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-09-16T17:10:29.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T17:10:29.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T17:10:29.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T17:10:29.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T17:10:29.798 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:10:29.791+0000 7ffa500b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:10:29.811 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:10:29.807+0000 7ffa500b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:10:29.813 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:10:29.807+0000 7ffa500b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:10:30.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T17:10:30.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:10:30.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:10:30.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:10:30.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:10:30.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:10:30.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:10:30.198 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:10:30.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:10:30.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:10:30.365 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:10:30.359+0000 7ffa500b87c0 -1 Falling back to public interface 2024-09-16T17:10:30.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:10:31.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:10:31.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:10:31.631 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:10:31.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:10:31.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:10:31.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:10:32.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:10:33.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:10:33.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:10:33.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:10:33.060 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:10:33.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:10:33.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:10:33.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:10:33.786 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:10:33.783+0000 7ffa500b87c0 -1 osd.0 25 log_to_monitors true 2024-09-16T17:10:34.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:10:34.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:10:34.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:10:34.601 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:10:34.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:10:34.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:10:35.342 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up out weight 0 up_from 31 up_thru 19 down_at 26 last_clean_interval [6,25) [v2:127.0.0.1:6802/2490504897,v1:127.0.0.1:6803/2490504897] [v2:127.0.0.1:6804/2490504897,v1:127.0.0.1:6805/2490504897] exists,up b395b92d-1ebb-48fe-a0fa-20cda55dae9c 2024-09-16T17:10:35.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:10:35.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:10:35.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:10:35.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:300: TEST_rep_backfill_unfound: ceph osd in 0 2024-09-16T17:10:37.096 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already in. 2024-09-16T17:10:37.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:302: TEST_rep_backfill_unfound: sleep 15 2024-09-16T17:10:52.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: seq 1 360 2024-09-16T17:10:52.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-16T17:10:52.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-16T17:10:52.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-16T17:10:52.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-16T17:10:52.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-16T17:10:52.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T17:10:52.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:52.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-16T17:10:52.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:52.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T17:10:52.559 INFO:tasks.workunit.client.0.smithi039.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:52.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-16T17:10:52.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-16T17:10:53.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-16T17:10:53.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-16T17:10:53.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-16T17:10:53.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-16T17:10:53.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-16T17:10:53.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T17:10:53.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:53.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:53.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-16T17:10:53.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T17:10:53.992 INFO:tasks.workunit.client.0.smithi039.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:53.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-16T17:10:53.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-16T17:10:54.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-16T17:10:54.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-16T17:10:54.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-16T17:10:54.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-16T17:10:54.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-16T17:10:54.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T17:10:55.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:55.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:55.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-16T17:10:55.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T17:10:55.434 INFO:tasks.workunit.client.0.smithi039.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:55.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-16T17:10:55.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-16T17:10:56.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-16T17:10:56.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-16T17:10:56.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-16T17:10:56.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-16T17:10:56.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-16T17:10:56.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T17:10:56.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:56.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:56.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-16T17:10:56.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T17:10:56.862 INFO:tasks.workunit.client.0.smithi039.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:56.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-16T17:10:56.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-16T17:10:57.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-16T17:10:57.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-16T17:10:57.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-16T17:10:57.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-16T17:10:57.865 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-16T17:10:57.865 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T17:10:58.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:58.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:58.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-16T17:10:58.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T17:10:58.282 INFO:tasks.workunit.client.0.smithi039.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:58.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-16T17:10:58.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-16T17:10:59.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-16T17:10:59.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-16T17:10:59.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-16T17:10:59.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-16T17:10:59.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-16T17:10:59.286 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T17:10:59.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:59.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:59.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-16T17:10:59.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T17:10:59.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-16T17:10:59.712 INFO:tasks.workunit.client.0.smithi039.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T17:10:59.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-16T17:11:00.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-16T17:11:00.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-16T17:11:00.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-16T17:11:00.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-16T17:11:00.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-16T17:11:00.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T17:11:01.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T17:11:01.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T17:11:01.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-16T17:11:01.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T17:11:01.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-16T17:11:01.155 INFO:tasks.workunit.client.0.smithi039.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T17:11:01.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-16T17:11:02.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-16T17:11:02.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-16T17:11:02.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-16T17:11:02.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-16T17:11:02.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-16T17:11:02.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T17:11:02.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T17:11:02.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T17:11:02.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-16T17:11:02.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T17:11:02.576 INFO:tasks.workunit.client.0.smithi039.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T17:11:02.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-16T17:11:02.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-16T17:11:03.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-16T17:11:03.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-16T17:11:03.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-16T17:11:03.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-16T17:11:03.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-16T17:11:03.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T17:11:04.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-09-16T17:11:04.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-09-16T17:11:04.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-16T17:11:04.007 INFO:tasks.workunit.client.0.smithi039.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-09-16T17:11:04.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 0 = 0 ']' 2024-09-16T17:11:04.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:308: TEST_rep_backfill_unfound: break 2024-09-16T17:11:04.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:314: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-09-16T17:11:04.409 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T17:11:04.409 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:10:37.972077+0000 0'0 39:61 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T19:44:20.476647+0000 0 0 2024-09-16T17:11:04.409 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:10:38.664709+0000 0'0 39:62 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:59:24.524704+0000 0 0 2024-09-16T17:11:04.409 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:10:39.086838+0000 0'0 39:53 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:33:44.648075+0000 0 0 2024-09-16T17:11:04.409 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:10:38.533709+0000 0'0 39:62 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:40:17.502661+0000 0 0 2024-09-16T17:11:04.409 INFO:tasks.workunit.client.0.smithi039.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-09-16T17:10:56.770678+0000 29'301 39:892 [2,1,0] 2 [2,1] 2 0'0 2024-09-16T17:09:28.850562+0000 0'0 2024-09-16T17:09:28.850562+0000 0 0 periodic scrub scheduled @ 2024-09-18T01:37:25.554128+0000 0 0 2024-09-16T17:11:04.409 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:10:37.821006+0000 0'0 39:61 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:36:53.381533+0000 0 0 2024-09-16T17:11:04.410 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:10:38.533695+0000 0'0 39:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-18T01:05:37.454052+0000 0 0 2024-09-16T17:11:04.410 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:10:39.086636+0000 0'0 39:53 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T21:48:41.965409+0000 0 0 2024-09-16T17:11:04.410 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:10:38.533571+0000 0'0 39:62 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:07:30.175542+0000 0'0 2024-09-16T17:07:30.175542+0000 0 0 periodic scrub scheduled @ 2024-09-17T21:38:49.426055+0000 0 0 2024-09-16T17:11:04.410 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T17:11:04.410 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T17:11:04.410 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T17:11:04.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-16T17:11:04.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: grep -q obj250 2024-09-16T17:11:04.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:318: TEST_rep_backfill_unfound: timeout 5 rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:09.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:319: TEST_rep_backfill_unfound: test 124 = 124 2024-09-16T17:11:09.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:321: TEST_rep_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-16T17:11:09.856 INFO:tasks.workunit.client.0.smithi039.stderr:pg has no unfound objects 2024-09-16T17:11:09.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:323: TEST_rep_backfill_unfound: wait_for_clean 2024-09-16T17:11:09.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:11:09.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:11:09.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:11:09.875 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:11:09.876 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:11:09.876 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:11:09.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:11:09.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:11:09.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:11:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:11:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:11:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:11:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:11:09.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:11:09.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:11:10.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:11:10.517 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:11:10.517 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:11:10.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:11:10.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:11:10.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:11:10.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986186 2024-09-16T17:11:10.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986186 2024-09-16T17:11:10.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-133143986186' 2024-09-16T17:11:10.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:11:10.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:11:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607595 2024-09-16T17:11:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607595 2024-09-16T17:11:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-133143986186 1-51539607595' 2024-09-16T17:11:10.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:11:10.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:11:11.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378658 2024-09-16T17:11:11.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378658 2024-09-16T17:11:11.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-133143986186 1-51539607595 2-81604378658' 2024-09-16T17:11:11.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:11:11.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-133143986186 2024-09-16T17:11:11.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:11:11.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:11:11.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-133143986186 2024-09-16T17:11:11.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:11:11.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986186 2024-09-16T17:11:11.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 133143986186' 2024-09-16T17:11:11.113 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 133143986186 2024-09-16T17:11:11.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:11:11.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986186 -lt 133143986186 2024-09-16T17:11:11.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:11:11.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607595 2024-09-16T17:11:11.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:11:11.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:11:11.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607595 2024-09-16T17:11:11.552 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:11:11.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607595 2024-09-16T17:11:11.553 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 51539607595 2024-09-16T17:11:11.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607595' 2024-09-16T17:11:11.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:11:12.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607593 -lt 51539607595 2024-09-16T17:11:12.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:11:13.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:11:13.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:11:13.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607595 -lt 51539607595 2024-09-16T17:11:13.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:11:13.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378658 2024-09-16T17:11:13.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:11:13.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:11:13.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378658 2024-09-16T17:11:13.546 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:11:13.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378658 2024-09-16T17:11:13.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378658' 2024-09-16T17:11:13.548 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 81604378658 2024-09-16T17:11:13.548 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:11:14.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378659 -lt 81604378658 2024-09-16T17:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:11:14.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:11:14.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:11:14.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:11:14.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:11:14.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:11:14.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:11:14.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:11:14.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:11:14.600 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:11:15.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:11:15.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:11:15.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:11:15.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:11:15.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:11:15.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-09-16T17:11:15.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T17:11:15.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-09-16T17:11:15.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:11:15.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T17:11:15.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:11:15.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:11:15.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:11:15.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:11:15.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:11:15.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:11:15.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:11:15.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:11:16.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:11:16.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:11:16.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:11:16.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:11:16.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-16T17:11:16.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-16T17:11:16.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T17:11:16.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T17:11:16.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T17:11:16.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T17:11:16.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T17:11:16.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T17:11:16.635 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T17:11:17.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T17:11:17.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T17:11:17.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-16T17:11:17.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:11:17.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T17:11:17.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:11:17.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:11:17.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:11:17.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:11:17.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:11:17.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:11:17.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:11:17.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:11:17.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-16T17:11:17.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:11:17.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:11:17.839 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:11:18.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-16T17:11:18.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:11:18.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:11:18.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: seq 1 300 2024-09-16T17:11:18.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj1 = obj250 ']' 2024-09-16T17:11:18.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj2 = obj250 ']' 2024-09-16T17:11:18.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj3 = obj250 ']' 2024-09-16T17:11:18.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj4 = obj250 ']' 2024-09-16T17:11:18.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj5 = obj250 ']' 2024-09-16T17:11:18.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj6 = obj250 ']' 2024-09-16T17:11:18.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj7 = obj250 ']' 2024-09-16T17:11:18.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj8 = obj250 ']' 2024-09-16T17:11:18.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj9 = obj250 ']' 2024-09-16T17:11:18.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj10 = obj250 ']' 2024-09-16T17:11:18.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj11 = obj250 ']' 2024-09-16T17:11:18.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj12 = obj250 ']' 2024-09-16T17:11:18.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj13 = obj250 ']' 2024-09-16T17:11:18.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj14 = obj250 ']' 2024-09-16T17:11:18.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj15 = obj250 ']' 2024-09-16T17:11:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj16 = obj250 ']' 2024-09-16T17:11:18.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj17 = obj250 ']' 2024-09-16T17:11:18.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj18 = obj250 ']' 2024-09-16T17:11:18.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:18.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:18.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj19 = obj250 ']' 2024-09-16T17:11:18.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj20 = obj250 ']' 2024-09-16T17:11:19.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj21 = obj250 ']' 2024-09-16T17:11:19.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj22 = obj250 ']' 2024-09-16T17:11:19.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj23 = obj250 ']' 2024-09-16T17:11:19.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj24 = obj250 ']' 2024-09-16T17:11:19.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj25 = obj250 ']' 2024-09-16T17:11:19.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj26 = obj250 ']' 2024-09-16T17:11:19.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj27 = obj250 ']' 2024-09-16T17:11:19.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj28 = obj250 ']' 2024-09-16T17:11:19.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj29 = obj250 ']' 2024-09-16T17:11:19.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj30 = obj250 ']' 2024-09-16T17:11:19.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj31 = obj250 ']' 2024-09-16T17:11:19.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj32 = obj250 ']' 2024-09-16T17:11:19.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj33 = obj250 ']' 2024-09-16T17:11:19.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj34 = obj250 ']' 2024-09-16T17:11:19.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj35 = obj250 ']' 2024-09-16T17:11:19.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj36 = obj250 ']' 2024-09-16T17:11:19.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj37 = obj250 ']' 2024-09-16T17:11:19.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj38 = obj250 ']' 2024-09-16T17:11:19.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj39 = obj250 ']' 2024-09-16T17:11:19.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj40 = obj250 ']' 2024-09-16T17:11:19.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj41 = obj250 ']' 2024-09-16T17:11:19.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj42 = obj250 ']' 2024-09-16T17:11:19.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj43 = obj250 ']' 2024-09-16T17:11:19.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj44 = obj250 ']' 2024-09-16T17:11:19.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj45 = obj250 ']' 2024-09-16T17:11:19.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj46 = obj250 ']' 2024-09-16T17:11:19.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj47 = obj250 ']' 2024-09-16T17:11:19.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj48 = obj250 ']' 2024-09-16T17:11:19.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:19.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:19.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj49 = obj250 ']' 2024-09-16T17:11:19.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj50 = obj250 ']' 2024-09-16T17:11:20.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj51 = obj250 ']' 2024-09-16T17:11:20.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj52 = obj250 ']' 2024-09-16T17:11:20.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj53 = obj250 ']' 2024-09-16T17:11:20.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj54 = obj250 ']' 2024-09-16T17:11:20.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj55 = obj250 ']' 2024-09-16T17:11:20.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj56 = obj250 ']' 2024-09-16T17:11:20.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj57 = obj250 ']' 2024-09-16T17:11:20.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj58 = obj250 ']' 2024-09-16T17:11:20.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj59 = obj250 ']' 2024-09-16T17:11:20.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj60 = obj250 ']' 2024-09-16T17:11:20.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj61 = obj250 ']' 2024-09-16T17:11:20.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj62 = obj250 ']' 2024-09-16T17:11:20.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj63 = obj250 ']' 2024-09-16T17:11:20.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj64 = obj250 ']' 2024-09-16T17:11:20.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj65 = obj250 ']' 2024-09-16T17:11:20.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj66 = obj250 ']' 2024-09-16T17:11:20.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj67 = obj250 ']' 2024-09-16T17:11:20.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj68 = obj250 ']' 2024-09-16T17:11:20.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj69 = obj250 ']' 2024-09-16T17:11:20.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj70 = obj250 ']' 2024-09-16T17:11:20.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj71 = obj250 ']' 2024-09-16T17:11:20.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj72 = obj250 ']' 2024-09-16T17:11:20.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj73 = obj250 ']' 2024-09-16T17:11:20.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj74 = obj250 ']' 2024-09-16T17:11:20.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj75 = obj250 ']' 2024-09-16T17:11:20.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj76 = obj250 ']' 2024-09-16T17:11:20.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj77 = obj250 ']' 2024-09-16T17:11:20.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj78 = obj250 ']' 2024-09-16T17:11:20.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj79 = obj250 ']' 2024-09-16T17:11:20.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj80 = obj250 ']' 2024-09-16T17:11:20.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj81 = obj250 ']' 2024-09-16T17:11:20.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:20.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:20.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj82 = obj250 ']' 2024-09-16T17:11:20.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj83 = obj250 ']' 2024-09-16T17:11:21.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj84 = obj250 ']' 2024-09-16T17:11:21.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj85 = obj250 ']' 2024-09-16T17:11:21.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj86 = obj250 ']' 2024-09-16T17:11:21.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj87 = obj250 ']' 2024-09-16T17:11:21.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj88 = obj250 ']' 2024-09-16T17:11:21.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj89 = obj250 ']' 2024-09-16T17:11:21.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj90 = obj250 ']' 2024-09-16T17:11:21.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj91 = obj250 ']' 2024-09-16T17:11:21.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj92 = obj250 ']' 2024-09-16T17:11:21.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj93 = obj250 ']' 2024-09-16T17:11:21.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj94 = obj250 ']' 2024-09-16T17:11:21.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj95 = obj250 ']' 2024-09-16T17:11:21.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj96 = obj250 ']' 2024-09-16T17:11:21.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj97 = obj250 ']' 2024-09-16T17:11:21.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj98 = obj250 ']' 2024-09-16T17:11:21.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj99 = obj250 ']' 2024-09-16T17:11:21.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj100 = obj250 ']' 2024-09-16T17:11:21.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj101 = obj250 ']' 2024-09-16T17:11:21.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj101 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj102 = obj250 ']' 2024-09-16T17:11:21.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj102 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj103 = obj250 ']' 2024-09-16T17:11:21.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj103 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj104 = obj250 ']' 2024-09-16T17:11:21.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj104 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj105 = obj250 ']' 2024-09-16T17:11:21.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj105 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj106 = obj250 ']' 2024-09-16T17:11:21.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj106 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj107 = obj250 ']' 2024-09-16T17:11:21.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj107 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj108 = obj250 ']' 2024-09-16T17:11:21.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj108 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj109 = obj250 ']' 2024-09-16T17:11:21.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj109 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj110 = obj250 ']' 2024-09-16T17:11:21.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj110 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj111 = obj250 ']' 2024-09-16T17:11:21.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj111 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj112 = obj250 ']' 2024-09-16T17:11:21.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj112 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:21.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:21.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj113 = obj250 ']' 2024-09-16T17:11:21.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj113 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj114 = obj250 ']' 2024-09-16T17:11:22.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj114 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj115 = obj250 ']' 2024-09-16T17:11:22.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj115 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj116 = obj250 ']' 2024-09-16T17:11:22.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj116 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj117 = obj250 ']' 2024-09-16T17:11:22.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj117 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj118 = obj250 ']' 2024-09-16T17:11:22.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj118 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj119 = obj250 ']' 2024-09-16T17:11:22.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj119 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj120 = obj250 ']' 2024-09-16T17:11:22.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj120 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj121 = obj250 ']' 2024-09-16T17:11:22.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj121 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj122 = obj250 ']' 2024-09-16T17:11:22.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj122 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj123 = obj250 ']' 2024-09-16T17:11:22.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj123 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj124 = obj250 ']' 2024-09-16T17:11:22.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj124 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj125 = obj250 ']' 2024-09-16T17:11:22.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj125 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj126 = obj250 ']' 2024-09-16T17:11:22.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj126 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj127 = obj250 ']' 2024-09-16T17:11:22.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj127 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj128 = obj250 ']' 2024-09-16T17:11:22.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj128 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj129 = obj250 ']' 2024-09-16T17:11:22.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj129 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj130 = obj250 ']' 2024-09-16T17:11:22.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj130 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj131 = obj250 ']' 2024-09-16T17:11:22.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj131 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj132 = obj250 ']' 2024-09-16T17:11:22.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj132 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj133 = obj250 ']' 2024-09-16T17:11:22.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj133 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj134 = obj250 ']' 2024-09-16T17:11:22.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj134 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj135 = obj250 ']' 2024-09-16T17:11:22.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj135 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj136 = obj250 ']' 2024-09-16T17:11:22.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj136 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj137 = obj250 ']' 2024-09-16T17:11:22.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj137 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj138 = obj250 ']' 2024-09-16T17:11:22.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj138 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj139 = obj250 ']' 2024-09-16T17:11:22.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj139 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj140 = obj250 ']' 2024-09-16T17:11:22.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj140 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj141 = obj250 ']' 2024-09-16T17:11:22.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj141 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj142 = obj250 ']' 2024-09-16T17:11:22.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj142 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj143 = obj250 ']' 2024-09-16T17:11:22.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj143 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj144 = obj250 ']' 2024-09-16T17:11:22.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj144 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj145 = obj250 ']' 2024-09-16T17:11:22.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj145 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:22.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:22.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj146 = obj250 ']' 2024-09-16T17:11:22.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj146 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj147 = obj250 ']' 2024-09-16T17:11:23.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj147 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj148 = obj250 ']' 2024-09-16T17:11:23.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj148 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj149 = obj250 ']' 2024-09-16T17:11:23.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj149 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj150 = obj250 ']' 2024-09-16T17:11:23.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj150 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj151 = obj250 ']' 2024-09-16T17:11:23.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj151 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj152 = obj250 ']' 2024-09-16T17:11:23.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj152 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj153 = obj250 ']' 2024-09-16T17:11:23.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj153 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj154 = obj250 ']' 2024-09-16T17:11:23.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj154 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj155 = obj250 ']' 2024-09-16T17:11:23.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj155 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj156 = obj250 ']' 2024-09-16T17:11:23.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj156 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj157 = obj250 ']' 2024-09-16T17:11:23.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj157 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj158 = obj250 ']' 2024-09-16T17:11:23.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj158 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj159 = obj250 ']' 2024-09-16T17:11:23.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj159 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj160 = obj250 ']' 2024-09-16T17:11:23.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj160 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj161 = obj250 ']' 2024-09-16T17:11:23.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj161 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj162 = obj250 ']' 2024-09-16T17:11:23.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj162 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj163 = obj250 ']' 2024-09-16T17:11:23.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj163 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj164 = obj250 ']' 2024-09-16T17:11:23.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj164 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj165 = obj250 ']' 2024-09-16T17:11:23.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj165 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj166 = obj250 ']' 2024-09-16T17:11:23.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj166 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj167 = obj250 ']' 2024-09-16T17:11:23.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj167 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj168 = obj250 ']' 2024-09-16T17:11:23.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj168 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj169 = obj250 ']' 2024-09-16T17:11:23.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj169 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj170 = obj250 ']' 2024-09-16T17:11:23.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj170 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj171 = obj250 ']' 2024-09-16T17:11:23.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj171 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj172 = obj250 ']' 2024-09-16T17:11:23.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj172 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj173 = obj250 ']' 2024-09-16T17:11:23.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj173 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj174 = obj250 ']' 2024-09-16T17:11:23.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj174 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj175 = obj250 ']' 2024-09-16T17:11:23.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj175 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj176 = obj250 ']' 2024-09-16T17:11:23.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj176 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:23.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:23.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj177 = obj250 ']' 2024-09-16T17:11:23.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj177 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj178 = obj250 ']' 2024-09-16T17:11:24.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj178 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj179 = obj250 ']' 2024-09-16T17:11:24.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj179 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj180 = obj250 ']' 2024-09-16T17:11:24.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj180 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj181 = obj250 ']' 2024-09-16T17:11:24.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj181 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj182 = obj250 ']' 2024-09-16T17:11:24.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj182 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj183 = obj250 ']' 2024-09-16T17:11:24.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj183 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj184 = obj250 ']' 2024-09-16T17:11:24.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj184 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj185 = obj250 ']' 2024-09-16T17:11:24.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj185 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj186 = obj250 ']' 2024-09-16T17:11:24.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj186 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj187 = obj250 ']' 2024-09-16T17:11:24.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj187 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj188 = obj250 ']' 2024-09-16T17:11:24.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj188 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj189 = obj250 ']' 2024-09-16T17:11:24.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj189 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj190 = obj250 ']' 2024-09-16T17:11:24.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj190 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj191 = obj250 ']' 2024-09-16T17:11:24.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj191 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj192 = obj250 ']' 2024-09-16T17:11:24.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj192 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj193 = obj250 ']' 2024-09-16T17:11:24.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj193 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj194 = obj250 ']' 2024-09-16T17:11:24.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj194 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj195 = obj250 ']' 2024-09-16T17:11:24.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj195 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj196 = obj250 ']' 2024-09-16T17:11:24.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj196 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj197 = obj250 ']' 2024-09-16T17:11:24.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj197 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj198 = obj250 ']' 2024-09-16T17:11:24.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj198 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj199 = obj250 ']' 2024-09-16T17:11:24.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj199 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj200 = obj250 ']' 2024-09-16T17:11:24.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj200 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj201 = obj250 ']' 2024-09-16T17:11:24.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj201 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj202 = obj250 ']' 2024-09-16T17:11:24.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj202 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj203 = obj250 ']' 2024-09-16T17:11:24.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj203 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj204 = obj250 ']' 2024-09-16T17:11:24.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj204 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj205 = obj250 ']' 2024-09-16T17:11:24.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj205 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj206 = obj250 ']' 2024-09-16T17:11:24.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj206 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj207 = obj250 ']' 2024-09-16T17:11:24.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj207 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj208 = obj250 ']' 2024-09-16T17:11:24.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj208 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:24.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:24.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj209 = obj250 ']' 2024-09-16T17:11:24.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj209 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj210 = obj250 ']' 2024-09-16T17:11:25.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj210 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj211 = obj250 ']' 2024-09-16T17:11:25.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj211 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj212 = obj250 ']' 2024-09-16T17:11:25.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj212 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj213 = obj250 ']' 2024-09-16T17:11:25.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj213 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj214 = obj250 ']' 2024-09-16T17:11:25.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj214 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj215 = obj250 ']' 2024-09-16T17:11:25.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj215 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj216 = obj250 ']' 2024-09-16T17:11:25.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj216 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj217 = obj250 ']' 2024-09-16T17:11:25.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj217 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj218 = obj250 ']' 2024-09-16T17:11:25.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj218 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj219 = obj250 ']' 2024-09-16T17:11:25.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj219 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj220 = obj250 ']' 2024-09-16T17:11:25.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj220 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj221 = obj250 ']' 2024-09-16T17:11:25.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj221 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj222 = obj250 ']' 2024-09-16T17:11:25.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj222 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj223 = obj250 ']' 2024-09-16T17:11:25.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj223 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj224 = obj250 ']' 2024-09-16T17:11:25.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj224 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj225 = obj250 ']' 2024-09-16T17:11:25.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj225 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj226 = obj250 ']' 2024-09-16T17:11:25.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj226 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj227 = obj250 ']' 2024-09-16T17:11:25.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj227 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj228 = obj250 ']' 2024-09-16T17:11:25.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj228 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj229 = obj250 ']' 2024-09-16T17:11:25.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj229 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj230 = obj250 ']' 2024-09-16T17:11:25.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj230 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj231 = obj250 ']' 2024-09-16T17:11:25.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj231 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj232 = obj250 ']' 2024-09-16T17:11:25.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj232 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj233 = obj250 ']' 2024-09-16T17:11:25.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj233 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj234 = obj250 ']' 2024-09-16T17:11:25.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj234 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj235 = obj250 ']' 2024-09-16T17:11:25.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj235 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj236 = obj250 ']' 2024-09-16T17:11:25.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj236 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj237 = obj250 ']' 2024-09-16T17:11:25.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj237 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj238 = obj250 ']' 2024-09-16T17:11:25.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj238 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:25.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:25.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj239 = obj250 ']' 2024-09-16T17:11:25.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj239 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj240 = obj250 ']' 2024-09-16T17:11:26.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj240 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj241 = obj250 ']' 2024-09-16T17:11:26.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj241 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj242 = obj250 ']' 2024-09-16T17:11:26.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj242 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj243 = obj250 ']' 2024-09-16T17:11:26.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj243 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj244 = obj250 ']' 2024-09-16T17:11:26.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj244 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj245 = obj250 ']' 2024-09-16T17:11:26.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj245 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj246 = obj250 ']' 2024-09-16T17:11:26.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj246 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj247 = obj250 ']' 2024-09-16T17:11:26.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj247 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj248 = obj250 ']' 2024-09-16T17:11:26.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj248 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj249 = obj250 ']' 2024-09-16T17:11:26.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj249 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj250 = obj250 ']' 2024-09-16T17:11:26.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:329: TEST_rep_backfill_unfound: rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.335 INFO:tasks.workunit.client.0.smithi039.stderr:error getting test-pool/obj250: (2) No such file or directory 2024-09-16T17:11:26.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj251 = obj250 ']' 2024-09-16T17:11:26.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj251 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj252 = obj250 ']' 2024-09-16T17:11:26.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj252 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj253 = obj250 ']' 2024-09-16T17:11:26.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj253 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj254 = obj250 ']' 2024-09-16T17:11:26.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj254 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj255 = obj250 ']' 2024-09-16T17:11:26.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj255 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj256 = obj250 ']' 2024-09-16T17:11:26.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj256 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj257 = obj250 ']' 2024-09-16T17:11:26.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj257 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj258 = obj250 ']' 2024-09-16T17:11:26.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj258 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj259 = obj250 ']' 2024-09-16T17:11:26.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj259 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj260 = obj250 ']' 2024-09-16T17:11:26.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj260 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj261 = obj250 ']' 2024-09-16T17:11:26.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj261 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj262 = obj250 ']' 2024-09-16T17:11:26.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj262 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj263 = obj250 ']' 2024-09-16T17:11:26.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj263 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj264 = obj250 ']' 2024-09-16T17:11:26.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj264 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj265 = obj250 ']' 2024-09-16T17:11:26.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj265 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj266 = obj250 ']' 2024-09-16T17:11:26.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj266 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj267 = obj250 ']' 2024-09-16T17:11:26.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj267 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj268 = obj250 ']' 2024-09-16T17:11:26.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj268 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj269 = obj250 ']' 2024-09-16T17:11:26.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj269 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj270 = obj250 ']' 2024-09-16T17:11:26.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj270 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj271 = obj250 ']' 2024-09-16T17:11:26.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj271 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:26.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:26.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj272 = obj250 ']' 2024-09-16T17:11:26.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj272 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj273 = obj250 ']' 2024-09-16T17:11:27.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj273 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj274 = obj250 ']' 2024-09-16T17:11:27.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj274 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj275 = obj250 ']' 2024-09-16T17:11:27.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj275 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj276 = obj250 ']' 2024-09-16T17:11:27.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj276 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj277 = obj250 ']' 2024-09-16T17:11:27.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj277 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj278 = obj250 ']' 2024-09-16T17:11:27.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj278 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj279 = obj250 ']' 2024-09-16T17:11:27.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj279 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj280 = obj250 ']' 2024-09-16T17:11:27.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj280 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj281 = obj250 ']' 2024-09-16T17:11:27.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj281 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj282 = obj250 ']' 2024-09-16T17:11:27.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj282 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj283 = obj250 ']' 2024-09-16T17:11:27.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj283 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj284 = obj250 ']' 2024-09-16T17:11:27.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj284 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj285 = obj250 ']' 2024-09-16T17:11:27.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj285 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj286 = obj250 ']' 2024-09-16T17:11:27.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj286 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj287 = obj250 ']' 2024-09-16T17:11:27.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj287 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj288 = obj250 ']' 2024-09-16T17:11:27.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj288 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj289 = obj250 ']' 2024-09-16T17:11:27.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj289 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj290 = obj250 ']' 2024-09-16T17:11:27.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj290 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj291 = obj250 ']' 2024-09-16T17:11:27.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj291 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj292 = obj250 ']' 2024-09-16T17:11:27.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj292 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj293 = obj250 ']' 2024-09-16T17:11:27.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj293 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj294 = obj250 ']' 2024-09-16T17:11:27.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj294 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj295 = obj250 ']' 2024-09-16T17:11:27.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj295 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj296 = obj250 ']' 2024-09-16T17:11:27.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj296 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj297 = obj250 ']' 2024-09-16T17:11:27.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj297 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj298 = obj250 ']' 2024-09-16T17:11:27.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj298 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj299 = obj250 ']' 2024-09-16T17:11:27.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj299 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:11:27.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj300 = obj250 ']' 2024-09-16T17:11:27.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj300 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:336: TEST_rep_backfill_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:11:27.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:338: TEST_rep_backfill_unfound: delete_pool test-pool 2024-09-16T17:11:27.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-09-16T17:11:27.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-09-16T17:11:28.455 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test-pool' does not exist 2024-09-16T17:11:28.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-16T17:11:28.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:11:28.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:11:28.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-16T17:11:28.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:11:28.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:11:28.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:11:28.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:11:28.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:11:28.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:11:28.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:11:28.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:11:28.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:11:28.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:11:28.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:11:28.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:11:28.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:11:28.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:11:28.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:11:28.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:11:28.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:11:28.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:11:28.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-16T17:11:28.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:11:28.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:11:28.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:11:28.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383662 2024-09-16T17:11:28.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:11:28.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:11:28.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-16T17:11:28.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-16T17:11:28.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:11:28.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-16T17:11:28.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:11:28.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:11:28.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-16T17:11:28.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:11:28.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:11:28.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:11:28.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:11:28.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:11:28.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:11:28.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:11:28.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:11:28.844 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:11:28.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:11:28.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:11:28.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:11:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:11:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:11:28.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:11:28.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:11:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:11:28.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:11:28.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-16T17:11:28.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:11:28.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:11:28.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:11:28.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383662 2024-09-16T17:11:28.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:11:28.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:11:28.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-16T17:11:28.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:11:28.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:11:28.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:11:28.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.383662 2024-09-16T17:11:28.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:11:28.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T17:11:28.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:11:28.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-16T17:11:28.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-16T17:11:28.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:11:28.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:11:28.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:11:28.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:11:28.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-16T17:11:28.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-16T17:11:29.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:11:29.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:11:29.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:11:29.244 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:11:29.244 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:11:29.244 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:11:29.245 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:11:29.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-16T17:11:29.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:11:29.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:11:29.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:11:29.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:11:29.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:11:29.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:11:29.491 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:11:29.491 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:11:29.491 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:11:29.492 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:11:29.492 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:11:29.492 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:11:29.492 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-mon.a.asok 2024-09-16T17:11:29.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:11:29.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383662/ceph-mon.a.asok config get fsid 2024-09-16T17:11:29.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:11:29.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:11:29.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:11:29.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:11:29.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:11:29.608 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:11:29.608 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:11:29.609 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:11:29.609 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:11:29.610 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:11:29.610 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:11:29.610 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-mon.a.asok 2024-09-16T17:11:29.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:11:29.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383662/ceph-mon.a.asok config get mon_host 2024-09-16T17:11:29.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-16T17:11:29.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:11:29.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:11:29.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:11:29.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:11:29.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-16T17:11:29.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:11:30.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:11:30.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:11:30.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:11:30.638 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:11:30.638 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:11:30.638 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:11:30.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:11:30.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:11:30.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:11:30.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-16T17:11:30.985 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T17:11:31.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_recovery_unfound td/osd-rep-recov-eio.sh 2024-09-16T17:11:31.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:343: TEST_rep_recovery_unfound: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:11:31.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:344: TEST_rep_recovery_unfound: local objname=myobject 2024-09-16T17:11:31.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:345: TEST_rep_recovery_unfound: local lastobj=100 2024-09-16T17:11:31.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:347: TEST_rep_recovery_unfound: local testobj=obj75 2024-09-16T17:11:31.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:349: TEST_rep_recovery_unfound: setup_osds 3 2024-09-16T17:11:31.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-09-16T17:11:31.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-16T17:11:31.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-16T17:11:31.004 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-09-16T17:11:31.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-09-16T17:11:31.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T17:11:31.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:11:31.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:11:31.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:11:31.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:11:31.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:11:31.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:11:31.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:11:31.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:11:31.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:11:31.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:11:31.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-16T17:11:31.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:11:31.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ed0497c-f09c-4c74-a562-afeb6dd8aa1f 2024-09-16T17:11:31.012 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 8ed0497c-f09c-4c74-a562-afeb6dd8aa1f 2024-09-16T17:11:31.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8ed0497c-f09c-4c74-a562-afeb6dd8aa1f' 2024-09-16T17:11:31.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:11:31.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDZuhm+p7SARAAM7hW4T+4cbJCJJR5csmBJA== 2024-09-16T17:11:31.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDZuhm+p7SARAAM7hW4T+4cbJCJJR5csmBJA=="}' 2024-09-16T17:11:31.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ed0497c-f09c-4c74-a562-afeb6dd8aa1f -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-16T17:11:31.438 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:11:31.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-16T17:11:31.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDZuhm+p7SARAAM7hW4T+4cbJCJJR5csmBJA== --osd-uuid 8ed0497c-f09c-4c74-a562-afeb6dd8aa1f 2024-09-16T17:11:31.487 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:31.486+0000 7f1e89fe37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:11:31.489 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:31.486+0000 7f1e89fe37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:11:31.491 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:31.490+0000 7f1e89fe37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:11:31.491 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:31.490+0000 7f1e89fe37c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T17:11:31.491 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:31.490+0000 7f1e89fe37c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T17:11:31.491 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:31.490+0000 7f1e89fe37c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-16T17:11:31.586 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:31.582+0000 7f1e89fe37c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-16T17:11:37.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-16T17:11:37.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:11:37.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:11:37.599 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:11:37.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:11:38.166 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:11:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:11:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:11:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:11:38.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:11:38.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:11:38.202 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:38.201+0000 7f4440d247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:11:38.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:38.217+0000 7f4440d247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:11:38.221 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:38.217+0000 7f4440d247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:11:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:11:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:11:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:11:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:11:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:11:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:38.587 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:11:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:11:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:38.776 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:38.773+0000 7f4440d247c0 -1 Falling back to public interface 2024-09-16T17:11:39.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:40.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:40.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:40.020 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:11:40.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:11:40.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:40.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:40.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:41.417 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:11:41.413+0000 7f4440d247c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:11:41.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:41.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:41.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:11:41.461 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:11:41.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:41.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:41.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:42.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:42.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:42.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:11:42.979 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:11:42.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:42.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:43.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:44.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:44.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:44.444 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:11:44.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:11:44.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:44.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:44.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:45.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:45.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:45.892 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:11:45.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:11:45.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:45.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:46.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:47.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:11:47.484 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:11:47.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:47.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:47.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:48.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:48.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:48.922 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:11:48.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:11:48.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:48.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:49.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:50.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:50.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:50.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:11:50.431 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:11:50.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:50.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:50.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:51.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:51.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:51.907 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:11:51.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:11:51.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:51.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:52.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:53.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:53.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:53.481 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:11:53.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:11:53.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:53.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:53.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:54.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:54.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:54.922 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:11:54.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:11:54.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:54.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:55.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:56.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:56.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:56.925 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:11:56.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:11:56.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:56.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:57.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:11:58.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:11:58.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:11:58.728 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:11:58.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:11:58.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:11:58.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:11:59.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:00.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:00.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:00.582 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:12:00.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:12:00.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:00.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:12:01.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:02.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:02.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:02.357 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:12:02.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:12:02.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:02.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:12:03.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:04.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:04.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:04.377 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:12:04.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:12:04.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:04.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:12:04.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:05.931 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:12:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:12:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:12:06.408 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3920487465,v1:127.0.0.1:6803/3920487465] [v2:127.0.0.1:6804/3920487465,v1:127.0.0.1:6805/3920487465] exists,up 8ed0497c-f09c-4c74-a562-afeb6dd8aa1f 2024-09-16T17:12:06.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:12:06.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:12:06.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:12:06.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:12:06.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:12:06.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:12:06.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:12:06.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:12:06.412 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:12:06.412 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:12:06.412 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:12:06.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:12:06.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:12:06.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2facdb71-fc26-4a1c-8033-92fd5cfb2324 2024-09-16T17:12:06.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2facdb71-fc26-4a1c-8033-92fd5cfb2324' 2024-09-16T17:12:06.415 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 2facdb71-fc26-4a1c-8033-92fd5cfb2324 2024-09-16T17:12:06.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:12:06.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmZuhmYwjBGRAAk8nHUqZynZ4cIyD3HjHt6g== 2024-09-16T17:12:06.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmZuhmYwjBGRAAk8nHUqZynZ4cIyD3HjHt6g=="}' 2024-09-16T17:12:06.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2facdb71-fc26-4a1c-8033-92fd5cfb2324 -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-16T17:12:07.377 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:12:07.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-16T17:12:07.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmZuhmYwjBGRAAk8nHUqZynZ4cIyD3HjHt6g== --osd-uuid 2facdb71-fc26-4a1c-8033-92fd5cfb2324 2024-09-16T17:12:07.443 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:07.441+0000 7f95edeb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:07.445 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:07.441+0000 7f95edeb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:07.448 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:07.445+0000 7f95edeb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:07.448 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:07.445+0000 7f95edeb67c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T17:12:07.448 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:07.445+0000 7f95edeb67c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T17:12:07.448 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:07.445+0000 7f95edeb67c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-16T17:12:07.518 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:07.513+0000 7f95edeb67c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-16T17:12:13.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-16T17:12:13.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:12:13.912 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T17:12:13.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T17:12:13.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:12:14.468 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:12:14.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T17:12:14.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:12:14.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:12:14.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:12:14.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:12:14.503 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:14.501+0000 7f9960c8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:14.525 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:14.521+0000 7f9960c8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:14.526 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:14.521+0000 7f9960c8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:14.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T17:12:14.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:12:14.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:12:14.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:12:14.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:12:14.890 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:12:14.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:14.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:12:14.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:14.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:15.072 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:15.069+0000 7f9960c8a7c0 -1 Falling back to public interface 2024-09-16T17:12:15.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:16.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:16.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:16.313 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:12:16.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:12:16.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:16.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:16.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:17.460 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:17.457+0000 7f9960c8a7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T17:12:17.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:17.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:17.746 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:12:17.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:12:17.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:17.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:18.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:19.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:19.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:19.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:12:19.309 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:12:19.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:19.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:19.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:20.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:20.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:20.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:12:20.755 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:12:20.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:20.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:21.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:22.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:22.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:22.265 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:12:22.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:12:22.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:22.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:22.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:23.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:23.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:23.702 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:12:23.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:12:23.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:23.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:24.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:25.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:25.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:25.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:12:25.219 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:12:25.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:25.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:25.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:26.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:26.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:26.751 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:12:26.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:12:26.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:26.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:27.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:28.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:28.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:28.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:12:28.283 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:12:28.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:28.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:28.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:29.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:29.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:29.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:12:29.742 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:12:29.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:29.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:30.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:31.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:31.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:31.180 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:12:31.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:12:31.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:31.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:32.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:33.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:33.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:33.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:12:33.065 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:12:33.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:33.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:33.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:34.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:34.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:34.860 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:12:34.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:12:34.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:34.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:35.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:36.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:36.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:36.785 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:12:36.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:12:36.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:36.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:37.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:38.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:38.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:38.653 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:12:38.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:12:38.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:38.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:39.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:40.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:40.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:40.731 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:12:40.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:12:40.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:40.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:41.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:42.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:42.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:42.524 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:12:42.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:12:42.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:42.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:12:43.142 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2737933267,v1:127.0.0.1:6811/2737933267] [v2:127.0.0.1:6812/2737933267,v1:127.0.0.1:6813/2737933267] exists,up 2facdb71-fc26-4a1c-8033-92fd5cfb2324 2024-09-16T17:12:43.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:12:43.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:12:43.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:12:43.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T17:12:43.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:12:43.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:12:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:12:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:12:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:12:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:12:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-16T17:12:43.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:12:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa27fea6-3c4a-4899-a9b5-90d920db1ec5 2024-09-16T17:12:43.149 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 aa27fea6-3c4a-4899-a9b5-90d920db1ec5 2024-09-16T17:12:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 aa27fea6-3c4a-4899-a9b5-90d920db1ec5' 2024-09-16T17:12:43.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:12:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALZ+hm+C0kChAAEiS4WnS1P7UvuayXKLW0Kg== 2024-09-16T17:12:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALZ+hm+C0kChAAEiS4WnS1P7UvuayXKLW0Kg=="}' 2024-09-16T17:12:43.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa27fea6-3c4a-4899-a9b5-90d920db1ec5 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-16T17:12:43.938 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:12:43.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-16T17:12:43.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALZ+hm+C0kChAAEiS4WnS1P7UvuayXKLW0Kg== --osd-uuid aa27fea6-3c4a-4899-a9b5-90d920db1ec5 2024-09-16T17:12:43.994 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:43.988+0000 7f917f0b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:44.028 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:44.024+0000 7f917f0b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:44.029 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:44.024+0000 7f917f0b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:44.030 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:44.024+0000 7f917f0b67c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T17:12:44.030 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:44.024+0000 7f917f0b67c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T17:12:44.030 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:44.024+0000 7f917f0b67c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-16T17:12:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:44.124+0000 7f917f0b67c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-16T17:12:50.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-16T17:12:50.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:12:50.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T17:12:50.315 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T17:12:50.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:12:50.882 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T17:12:50.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T17:12:50.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:12:50.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:12:50.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:12:50.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:12:50.917 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:50.912+0000 7f9d761c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:50.959 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:50.956+0000 7f9d761c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:50.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:50.956+0000 7f9d761c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:12:51.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T17:12:51.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:12:51.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:12:51.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:12:51.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:12:51.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:51.279 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:12:51.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:12:51.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:51.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:12:51.509 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:51.504+0000 7f9d761c57c0 -1 Falling back to public interface 2024-09-16T17:12:51.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:52.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:52.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:52.721 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:12:52.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:12:52.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:52.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:12:53.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:54.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:54.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:54.151 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:12:54.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:12:54.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:54.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:12:54.270 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:12:54.264+0000 7f9d761c57c0 -1 osd.2 0 log_to_monitors true 2024-09-16T17:12:54.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:55.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:55.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:55.662 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:12:55.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:12:55.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:55.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:12:56.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:57.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:57.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:57.246 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:12:57.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:12:57.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:57.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:12:57.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:12:58.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:12:58.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:12:58.747 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:12:58.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:12:58.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:12:58.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:12:59.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:00.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:00.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:00.280 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:13:00.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:13:00.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:00.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:00.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:01.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:01.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:01.902 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:13:01.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:13:01.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:01.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:02.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:03.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:03.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:03.492 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:13:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:13:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:03.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:04.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:05.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:05.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:05.120 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:13:05.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:13:05.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:05.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:05.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:06.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:06.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:06.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:13:06.606 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:13:06.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:06.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:07.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:08.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:08.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:08.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:13:08.170 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:13:08.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:08.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:08.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:09.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:09.745 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:13:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:13:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:09.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:10.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:11.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:11.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:11.747 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:13:11.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:13:11.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:11.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:12.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:13.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:13.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:13.746 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:13:13.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:13:13.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:13.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:14.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:15.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:15.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:15.480 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:13:15.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:13:15.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:15.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:16.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:17.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:17.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:17.497 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:13:17.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:13:17.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:17.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:17.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:18.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:18.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:18.986 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:13:18.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:13:18.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:18.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:19.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:20.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:20.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:20.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T17:13:20.756 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:13:20.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:20.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:13:21.236 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3968829268,v1:127.0.0.1:6819/3968829268] [v2:127.0.0.1:6820/3968829268,v1:127.0.0.1:6821/3968829268] exists,up aa27fea6-3c4a-4899-a9b5-90d920db1ec5 2024-09-16T17:13:21.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:13:21.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:13:21.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:13:21.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-16T17:13:21.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:13:21.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:13:21.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:13:21.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:13:21.238 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:13:21.238 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:13:21.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:13:21.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:13:21.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:13:21.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:13:21.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:13:21.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:13:21.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:13:21.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:13:21.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:13:21.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:13:21.982 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:13:21.982 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:13:21.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:13:21.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:13:21.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:13:22.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803798 2024-09-16T17:13:22.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803798 2024-09-16T17:13:22.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803798' 2024-09-16T17:13:22.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:13:22.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:13:22.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574862 2024-09-16T17:13:22.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574862 2024-09-16T17:13:22.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803798 1-55834574862' 2024-09-16T17:13:22.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:13:22.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:13:22.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345932 2024-09-16T17:13:22.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345932 2024-09-16T17:13:22.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803798 1-55834574862 2-85899345932' 2024-09-16T17:13:22.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:13:22.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803798 2024-09-16T17:13:22.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:13:22.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:13:22.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803798 2024-09-16T17:13:22.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:13:22.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803798 2024-09-16T17:13:22.620 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803798 2024-09-16T17:13:22.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803798' 2024-09-16T17:13:22.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:13:23.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803795 -lt 25769803798 2024-09-16T17:13:23.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:13:24.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:13:24.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:13:24.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803798 -lt 25769803798 2024-09-16T17:13:24.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:13:24.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574862 2024-09-16T17:13:24.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:13:24.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:13:24.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574862 2024-09-16T17:13:24.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:13:24.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574862 2024-09-16T17:13:24.688 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 55834574862 2024-09-16T17:13:24.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574862' 2024-09-16T17:13:24.689 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:13:25.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574863 -lt 55834574862 2024-09-16T17:13:25.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:13:25.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-85899345932 2024-09-16T17:13:25.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:13:25.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:13:25.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-85899345932 2024-09-16T17:13:25.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:13:25.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345932 2024-09-16T17:13:25.167 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 85899345932 2024-09-16T17:13:25.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 85899345932' 2024-09-16T17:13:25.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:13:25.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345932 -lt 85899345932 2024-09-16T17:13:25.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:13:25.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:13:25.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:13:26.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T17:13:26.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:13:26.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:13:26.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:13:26.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:13:26.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:13:26.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:13:26.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:13:26.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T17:13:26.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:13:26.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:13:26.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:13:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 8 2024-09-16T17:13:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-09-16T17:13:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T17:13:27.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-09-16T17:13:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:13:27.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T17:13:27.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:13:27.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:13:27.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:13:27.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:13:27.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:13:27.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:13:27.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:13:27.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:13:27.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T17:13:27.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:13:27.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:13:27.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:13:28.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T17:13:28.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:13:28.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:13:28.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:351: TEST_rep_recovery_unfound: local poolname=test-pool 2024-09-16T17:13:28.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:352: TEST_rep_recovery_unfound: create_pool test-pool 1 1 2024-09-16T17:13:28.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-09-16T17:13:28.869 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test-pool' already exists 2024-09-16T17:13:28.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T17:13:29.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:353: TEST_rep_recovery_unfound: wait_for_clean 2024-09-16T17:13:29.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:13:29.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:13:29.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:13:29.889 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:13:29.890 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:13:29.891 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:13:29.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:13:29.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:13:29.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:13:30.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:13:30.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:13:30.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:13:30.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:13:30.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:13:30.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:13:30.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:13:30.454 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:13:30.454 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:13:30.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:13:30.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:13:30.454 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:13:30.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803801 2024-09-16T17:13:30.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803801 2024-09-16T17:13:30.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803801' 2024-09-16T17:13:30.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:13:30.653 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:13:30.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574866 2024-09-16T17:13:30.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574866 2024-09-16T17:13:30.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803801 1-55834574866' 2024-09-16T17:13:30.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:13:30.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:13:30.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345936 2024-09-16T17:13:30.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345936 2024-09-16T17:13:30.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803801 1-55834574866 2-85899345936' 2024-09-16T17:13:30.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:13:30.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:13:30.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803801 2024-09-16T17:13:30.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:13:30.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803801 2024-09-16T17:13:30.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:13:30.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803801 2024-09-16T17:13:30.976 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 25769803801 2024-09-16T17:13:30.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2024-09-16T17:13:30.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:13:31.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803799 -lt 25769803801 2024-09-16T17:13:31.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:13:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:13:32.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:13:32.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803801 -lt 25769803801 2024-09-16T17:13:32.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:13:32.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574866 2024-09-16T17:13:32.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:13:32.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:13:32.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574866 2024-09-16T17:13:32.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:13:32.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574866 2024-09-16T17:13:32.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574866' 2024-09-16T17:13:32.818 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 55834574866 2024-09-16T17:13:32.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:13:33.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574866 -lt 55834574866 2024-09-16T17:13:33.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:13:33.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-85899345936 2024-09-16T17:13:33.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:13:33.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:13:33.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-85899345936 2024-09-16T17:13:33.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:13:33.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345936 2024-09-16T17:13:33.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 85899345936' 2024-09-16T17:13:33.280 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 85899345936 2024-09-16T17:13:33.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:13:33.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345936 -lt 85899345936 2024-09-16T17:13:33.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:13:33.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:13:33.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:13:34.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:13:34.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:13:34.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:13:34.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:13:34.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:13:34.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:13:34.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:13:34.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:13:34.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-16T17:13:34.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:13:34.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:13:34.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:13:35.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-16T17:13:35.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:13:35.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:13:35.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:355: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-09-16T17:13:35.716 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T17:13:35.716 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:22.490074+0000 0'0 25:27 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-18T02:42:02.974583+0000 0 0 2024-09-16T17:13:35.716 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:22.491601+0000 0'0 25:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:06:50.829908+0000 0 0 2024-09-16T17:13:35.717 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:23.086178+0000 0'0 25:15 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:16:16.611871+0000 0 0 2024-09-16T17:13:35.717 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:22.491855+0000 0'0 25:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T20:26:29.910593+0000 0 0 2024-09-16T17:13:35.717 INFO:tasks.workunit.client.0.smithi039.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:30.298417+0000 0'0 26:11 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-16T17:13:28.647300+0000 0'0 2024-09-16T17:13:28.647300+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:44:39.768960+0000 0 0 2024-09-16T17:13:35.717 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:22.491099+0000 0'0 25:27 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:02:29.205812+0000 0 0 2024-09-16T17:13:35.717 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:23.231476+0000 0'0 25:37 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:23:46.774943+0000 0 0 2024-09-16T17:13:35.717 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:23.231603+0000 0'0 25:15 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:55:30.406469+0000 0 0 2024-09-16T17:13:35.717 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:22.491479+0000 0'0 25:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:57:14.825537+0000 0 0 2024-09-16T17:13:35.717 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T17:13:35.717 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T17:13:35.717 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T17:13:35.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:357: TEST_rep_recovery_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-09-16T17:13:35.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:13:35.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-09-16T17:13:35.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-09-16T17:13:35.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:13:35.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-16T17:13:35.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:13:35.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-16T17:13:35.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:13:35.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-16T17:13:35.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T17:13:35.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-16T17:13:35.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:35.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: get_osds test-pool myobject 2024-09-16T17:13:35.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-16T17:13:35.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T17:13:35.908 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-09-16T17:13:35.908 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:13:36.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-16T17:13:36.329 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:13:36.329 INFO:tasks.workunit.client.0.smithi039.stderr:0' 2024-09-16T17:13:36.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-09-16T17:13:36.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: initial_osds=('2' '1' '0') 2024-09-16T17:13:36.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: local -a initial_osds 2024-09-16T17:13:36.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:360: TEST_rep_recovery_unfound: local last_osd=0 2024-09-16T17:13:36.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:361: TEST_rep_recovery_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-09-16T17:13:36.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:13:36.330 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:13:36.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:13:36.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:13:36.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:13:36.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:13:36.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:362: TEST_rep_recovery_unfound: ceph osd down 0 2024-09-16T17:13:37.640 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-16T17:13:37.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:363: TEST_rep_recovery_unfound: ceph osd out 0 2024-09-16T17:13:38.281 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-16T17:13:38.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:365: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-09-16T17:13:38.701 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T17:13:38.701 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:13:37.554085+0000 0'0 29:32 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:51:11.137035+0000 0 0 2024-09-16T17:13:38.701 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:13:37.553471+0000 0'0 29:32 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-18T02:37:09.026912+0000 0 0 2024-09-16T17:13:38.701 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:23.086178+0000 0'0 25:15 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:16:16.611871+0000 0 0 2024-09-16T17:13:38.701 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:13:37.553563+0000 0'0 29:32 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:35:31.236563+0000 0 0 2024-09-16T17:13:38.701 INFO:tasks.workunit.client.0.smithi039.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:30.298417+0000 0'0 26:11 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-16T17:13:28.647300+0000 0'0 2024-09-16T17:13:28.647300+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:44:39.768960+0000 0 0 2024-09-16T17:13:38.701 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:13:37.553693+0000 0'0 29:32 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:31:29.230520+0000 0 0 2024-09-16T17:13:38.702 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:13:37.553563+0000 0'0 29:31 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T22:48:36.720573+0000 0 0 2024-09-16T17:13:38.702 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:13:23.231603+0000 0'0 25:15 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:55:30.406469+0000 0 0 2024-09-16T17:13:38.702 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-16T17:13:37.553475+0000 0'0 29:32 [1,2] 1 [1,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:57:41.837833+0000 0 0 2024-09-16T17:13:38.702 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T17:13:38.702 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T17:13:38.702 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T17:13:38.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:367: TEST_rep_recovery_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-09-16T17:13:38.720 INFO:tasks.workunit.client.0.smithi039.stderr:4+0 records in 2024-09-16T17:13:38.720 INFO:tasks.workunit.client.0.smithi039.stderr:4+0 records out 2024-09-16T17:13:38.720 INFO:tasks.workunit.client.0.smithi039.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000175531 s, 23.3 MB/s 2024-09-16T17:13:38.721 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: seq 1 100 2024-09-16T17:13:38.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:38.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:38.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:38.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:39.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:39.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:39.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:39.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:39.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:39.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:39.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:39.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:39.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:39.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:39.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:39.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:40.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:40.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:40.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:40.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:40.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:40.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:40.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:40.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:40.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:40.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:40.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:41.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:41.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:41.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:41.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:41.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:41.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:41.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:41.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:41.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:41.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:41.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:41.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:42.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:42.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:42.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:42.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:42.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:42.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:42.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:42.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:42.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:42.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:42.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:42.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:42.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:42.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:43.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:43.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:43.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:43.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:43.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:43.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:43.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:43.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:43.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:43.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:44.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:44.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:44.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:44.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:44.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:44.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:44.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:44.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:44.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:44.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:44.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:44.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:44.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:44.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:45.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:45.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:45.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:45.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:45.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:45.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:45.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:45.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:45.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:45.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:45.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:46.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:46.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:46.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:46.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:46.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:46.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:46.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:46.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:46.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:46.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:46.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:46.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:47.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:47.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:47.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:47.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:47.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:47.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:47.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:47.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:47.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:47.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:47.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:47.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:48.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:48.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:48.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:48.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:48.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:48.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:48.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:48.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:48.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:48.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:48.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:48.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:49.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:49.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:49.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:49.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:49.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:49.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:49.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:49.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:49.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:49.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:49.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:49.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:49.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:50.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:50.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:50.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:50.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:50.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:50.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:50.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:50.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:50.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:50.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:50.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:50.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:51.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:51.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:51.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:51.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:51.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:51.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:51.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:51.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:51.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:51.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:51.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:51.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:52.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:52.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:52.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:52.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:52.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:52.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:52.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:52.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:52.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:52.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:52.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:52.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:52.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:53.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:53.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:53.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:53.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:53.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:53.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:53.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:53.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:53.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:53.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:54.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:54.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:54.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:54.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:54.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:54.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:54.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:54.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:54.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:54.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:54.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:13:54.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-16T17:13:54.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:373: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 0 2024-09-16T17:13:54.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:13:54.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=test-pool 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj75 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds test-pool obj75 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-16T17:13:54.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-16T17:13:54.963 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-09-16T17:13:54.963 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:13:55.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-16T17:13:55.401 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-09-16T17:13:55.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-16T17:13:55.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('2' '1') 2024-09-16T17:13:55.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:13:55.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=2 2024-09-16T17:13:55.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:13:55.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:13:55.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-09-16T17:13:55.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:13:55.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-16T17:13:55.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:13:55.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-09-16T17:13:55.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:13:55.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:13:55.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:13:55.404 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-09-16T17:13:55.404 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T17:13:55.404 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T17:13:55.405 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:13:55.405 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:13:55.405 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:13:55.405 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.2.asok 2024-09-16T17:13:55.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:13:55.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:13:55.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:13:55.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:13:55.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.2 2024-09-16T17:13:55.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T17:13:55.533 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T17:13:55.534 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:13:55.534 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:13:55.534 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:13:55.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.2.asok 2024-09-16T17:13:55.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:13:55.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.2.asok injectdataerr test-pool obj75 2024-09-16T17:13:55.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:374: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 1 2024-09-16T17:13:55.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2024-09-16T17:13:55.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T17:13:55.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T17:13:55.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T17:13:55.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=test-pool 2024-09-16T17:13:55.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T17:13:55.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj75 2024-09-16T17:13:55.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T17:13:55.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:13:55.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T17:13:55.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T17:13:55.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T17:13:55.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds test-pool obj75 2024-09-16T17:13:55.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-16T17:13:55.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-16T17:13:55.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-09-16T17:13:55.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T17:13:56.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-16T17:13:56.100 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-09-16T17:13:56.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-16T17:13:56.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('2' '1') 2024-09-16T17:13:56.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T17:13:56.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T17:13:56.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2024-09-16T17:13:56.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2024-09-16T17:13:56.100 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-16T17:13:56.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T17:13:56.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T17:13:56.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T17:13:56.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T17:13:56.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T17:13:56.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T17:13:56.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T17:13:56.102 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T17:13:56.102 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:13:56.102 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:13:56.103 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:13:56.103 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:13:56.103 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:13:56.103 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:13:56.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383662/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T17:13:56.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T17:13:56.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T17:13:56.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T17:13:56.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T17:13:56.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T17:13:56.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T17:13:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:13:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:13:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:13:56.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383662/ceph-osd.1.asok 2024-09-16T17:13:56.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T17:13:56.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383662/ceph-osd.1.asok injectdataerr test-pool obj75 2024-09-16T17:13:56.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:376: TEST_rep_recovery_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-09-16T17:13:56.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:13:56.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T17:13:56.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T17:13:56.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T17:13:56.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-16T17:13:56.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T17:13:56.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:13:56.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:13:56.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:13:56.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-16T17:13:56.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-16T17:13:56.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T17:13:56.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T17:13:56.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-16T17:13:56.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T17:13:56.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:13:56.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:13:56.349 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:13:56.349 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T17:13:56.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-16T17:13:56.351 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:13:56.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T17:13:56.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=002d35f1-7609-4332-991c-0467e7c6bcdf --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383662/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:13:56.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-09-16T17:13:56.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T17:13:56.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T17:13:56.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T17:13:56.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T17:13:56.378 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:13:56.379+0000 7f6c654ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:13:56.381 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:13:56.379+0000 7f6c654ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:13:56.382 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:13:56.383+0000 7f6c654ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:13:56.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T17:13:56.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:13:56.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:13:56.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:13:56.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:13:56.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:56.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:13:56.783 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:13:56.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:56.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:13:56.940 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:13:56.939+0000 7f6c654ca7c0 -1 Falling back to public interface 2024-09-16T17:13:57.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:58.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:58.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:58.219 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:13:58.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:13:58.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:58.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:13:58.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:13:59.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:13:59.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:13:59.641 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:13:59.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:13:59.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:13:59.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:00.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:14:00.261 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:00.259+0000 7f6c654ca7c0 -1 osd.0 26 log_to_monitors true 2024-09-16T17:14:01.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:14:01.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:01.112 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:14:01.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:14:01.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:01.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:01.752 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up out weight 0 up_from 32 up_thru 21 down_at 27 last_clean_interval [6,26) [v2:127.0.0.1:6802/198298487,v1:127.0.0.1:6803/198298487] [v2:127.0.0.1:6804/198298487,v1:127.0.0.1:6805/198298487] exists,up 8ed0497c-f09c-4c74-a562-afeb6dd8aa1f 2024-09-16T17:14:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:14:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:14:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:14:01.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:377: TEST_rep_recovery_unfound: ceph osd in 0 2024-09-16T17:14:03.158 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already in. 2024-09-16T17:14:03.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:379: TEST_rep_recovery_unfound: sleep 15 2024-09-16T17:14:18.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: seq 1 100 2024-09-16T17:14:18.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2024-09-16T17:14:18.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2024-09-16T17:14:18.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-16T17:14:18.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-16T17:14:18.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-16T17:14:18.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T17:14:18.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovery_unfound+degraded 2024-09-16T17:14:18.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2024-09-16T17:14:18.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovery_unfound+degraded 2024-09-16T17:14:18.611 INFO:tasks.workunit.client.0.smithi039.stdout:active+recovery_unfound+degraded 2024-09-16T17:14:18.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 0 = 0 ']' 2024-09-16T17:14:18.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:385: TEST_rep_recovery_unfound: break 2024-09-16T17:14:18.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:391: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-09-16T17:14:18.999 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T17:14:18.999 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:14:04.086443+0000 0'0 39:59 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:59:47.428590+0000 0 0 2024-09-16T17:14:18.999 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:14:05.394634+0000 0'0 40:60 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T21:56:01.848567+0000 0 0 2024-09-16T17:14:18.999 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:14:04.900834+0000 0'0 39:47 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-18T01:37:22.750969+0000 0 0 2024-09-16T17:14:19.000 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:14:05.394847+0000 0'0 40:60 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-18T03:52:13.469631+0000 0 0 2024-09-16T17:14:19.000 INFO:tasks.workunit.client.0.smithi039.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-09-16T17:14:09.404423+0000 30'101 40:343 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-16T17:13:28.647300+0000 0'0 2024-09-16T17:13:28.647300+0000 0 0 periodic scrub scheduled @ 2024-09-18T02:24:03.647130+0000 0 0 2024-09-16T17:14:19.000 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:14:04.086207+0000 0'0 39:59 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:57:34.001715+0000 0 0 2024-09-16T17:14:19.000 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:14:04.756727+0000 0'0 39:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-18T04:50:13.754799+0000 0 0 2024-09-16T17:14:19.000 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:14:04.899571+0000 0'0 39:47 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:28:30.727867+0000 0 0 2024-09-16T17:14:19.000 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T17:14:05.394788+0000 0'0 40:60 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T17:11:30.859339+0000 0'0 2024-09-16T17:11:30.859339+0000 0 0 periodic scrub scheduled @ 2024-09-17T23:19:09.854989+0000 0 0 2024-09-16T17:14:19.000 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T17:14:19.000 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T17:14:19.000 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-16T17:14:19.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-16T17:14:19.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: grep -q obj75 2024-09-16T17:14:19.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:395: TEST_rep_recovery_unfound: timeout 5 rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:24.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:396: TEST_rep_recovery_unfound: test 124 = 124 2024-09-16T17:14:24.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:398: TEST_rep_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-16T17:14:24.444 INFO:tasks.workunit.client.0.smithi039.stderr:pg has no unfound objects 2024-09-16T17:14:24.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:400: TEST_rep_recovery_unfound: wait_for_clean 2024-09-16T17:14:24.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:14:24.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:14:24.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:14:24.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:14:24.462 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:14:24.462 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:14:24.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:14:24.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:14:24.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:14:24.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:14:24.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:14:24.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:14:24.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:14:24.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:14:24.583 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:14:25.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:14:25.021 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:14:25.021 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:14:25.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:14:25.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:14:25.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:14:25.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=137438953480 2024-09-16T17:14:25.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 137438953480 2024-09-16T17:14:25.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-137438953480' 2024-09-16T17:14:25.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:14:25.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:14:25.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574879 2024-09-16T17:14:25.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574879 2024-09-16T17:14:25.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-137438953480 1-55834574879' 2024-09-16T17:14:25.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:14:25.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:14:25.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345948 2024-09-16T17:14:25.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345948 2024-09-16T17:14:25.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-137438953480 1-55834574879 2-85899345948' 2024-09-16T17:14:25.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:14:25.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-137438953480 2024-09-16T17:14:25.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:14:25.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:14:25.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-137438953480 2024-09-16T17:14:25.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:14:25.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=137438953480 2024-09-16T17:14:25.531 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 137438953480 2024-09-16T17:14:25.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 137438953480' 2024-09-16T17:14:25.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:14:25.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 137438953480 -lt 137438953480 2024-09-16T17:14:25.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:14:25.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574879 2024-09-16T17:14:25.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:14:25.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:14:25.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574879 2024-09-16T17:14:25.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:14:25.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574879 2024-09-16T17:14:25.991 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 55834574879 2024-09-16T17:14:25.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574879' 2024-09-16T17:14:25.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:14:26.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574879 -lt 55834574879 2024-09-16T17:14:26.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:14:26.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-85899345948 2024-09-16T17:14:26.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:14:26.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:14:26.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:14:26.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-85899345948 2024-09-16T17:14:26.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345948 2024-09-16T17:14:26.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 85899345948' 2024-09-16T17:14:26.419 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 85899345948 2024-09-16T17:14:26.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:14:26.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345946 -lt 85899345948 2024-09-16T17:14:26.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:14:27.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:14:27.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:14:28.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345949 -lt 85899345948 2024-09-16T17:14:28.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:14:28.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:14:28.263 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:14:28.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-16T17:14:28.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:14:28.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:14:28.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:14:28.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:14:28.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:14:28.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:14:28.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:14:29.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-16T17:14:29.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:14:29.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:14:29.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:14:29.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-16T17:14:29.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:14:29.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:14:29.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: seq 1 100 2024-09-16T17:14:29.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:29.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj1 = obj75 ']' 2024-09-16T17:14:29.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:29.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:29.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:29.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj2 = obj75 ']' 2024-09-16T17:14:29.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:29.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:29.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:29.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj3 = obj75 ']' 2024-09-16T17:14:29.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj4 = obj75 ']' 2024-09-16T17:14:30.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj5 = obj75 ']' 2024-09-16T17:14:30.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj6 = obj75 ']' 2024-09-16T17:14:30.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj7 = obj75 ']' 2024-09-16T17:14:30.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj8 = obj75 ']' 2024-09-16T17:14:30.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj9 = obj75 ']' 2024-09-16T17:14:30.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj10 = obj75 ']' 2024-09-16T17:14:30.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj11 = obj75 ']' 2024-09-16T17:14:30.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj12 = obj75 ']' 2024-09-16T17:14:30.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj13 = obj75 ']' 2024-09-16T17:14:30.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj14 = obj75 ']' 2024-09-16T17:14:30.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj15 = obj75 ']' 2024-09-16T17:14:30.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj16 = obj75 ']' 2024-09-16T17:14:30.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj17 = obj75 ']' 2024-09-16T17:14:30.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj18 = obj75 ']' 2024-09-16T17:14:30.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj19 = obj75 ']' 2024-09-16T17:14:30.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj20 = obj75 ']' 2024-09-16T17:14:30.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj21 = obj75 ']' 2024-09-16T17:14:30.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj22 = obj75 ']' 2024-09-16T17:14:30.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj23 = obj75 ']' 2024-09-16T17:14:30.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj24 = obj75 ']' 2024-09-16T17:14:30.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj25 = obj75 ']' 2024-09-16T17:14:30.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj26 = obj75 ']' 2024-09-16T17:14:30.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj27 = obj75 ']' 2024-09-16T17:14:30.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj28 = obj75 ']' 2024-09-16T17:14:30.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj29 = obj75 ']' 2024-09-16T17:14:30.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj30 = obj75 ']' 2024-09-16T17:14:30.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj31 = obj75 ']' 2024-09-16T17:14:30.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj32 = obj75 ']' 2024-09-16T17:14:30.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj33 = obj75 ']' 2024-09-16T17:14:30.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:30.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:30.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj34 = obj75 ']' 2024-09-16T17:14:30.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj35 = obj75 ']' 2024-09-16T17:14:31.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj36 = obj75 ']' 2024-09-16T17:14:31.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj37 = obj75 ']' 2024-09-16T17:14:31.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj38 = obj75 ']' 2024-09-16T17:14:31.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj39 = obj75 ']' 2024-09-16T17:14:31.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj40 = obj75 ']' 2024-09-16T17:14:31.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj41 = obj75 ']' 2024-09-16T17:14:31.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj42 = obj75 ']' 2024-09-16T17:14:31.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj43 = obj75 ']' 2024-09-16T17:14:31.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj44 = obj75 ']' 2024-09-16T17:14:31.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj45 = obj75 ']' 2024-09-16T17:14:31.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj46 = obj75 ']' 2024-09-16T17:14:31.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj47 = obj75 ']' 2024-09-16T17:14:31.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj48 = obj75 ']' 2024-09-16T17:14:31.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj49 = obj75 ']' 2024-09-16T17:14:31.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj50 = obj75 ']' 2024-09-16T17:14:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj51 = obj75 ']' 2024-09-16T17:14:31.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj52 = obj75 ']' 2024-09-16T17:14:31.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj53 = obj75 ']' 2024-09-16T17:14:31.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj54 = obj75 ']' 2024-09-16T17:14:31.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj55 = obj75 ']' 2024-09-16T17:14:31.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj56 = obj75 ']' 2024-09-16T17:14:31.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj57 = obj75 ']' 2024-09-16T17:14:31.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj58 = obj75 ']' 2024-09-16T17:14:31.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj59 = obj75 ']' 2024-09-16T17:14:31.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj60 = obj75 ']' 2024-09-16T17:14:31.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj61 = obj75 ']' 2024-09-16T17:14:31.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj62 = obj75 ']' 2024-09-16T17:14:31.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj63 = obj75 ']' 2024-09-16T17:14:31.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj64 = obj75 ']' 2024-09-16T17:14:31.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:31.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:31.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj65 = obj75 ']' 2024-09-16T17:14:31.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj66 = obj75 ']' 2024-09-16T17:14:32.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj67 = obj75 ']' 2024-09-16T17:14:32.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj68 = obj75 ']' 2024-09-16T17:14:32.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj69 = obj75 ']' 2024-09-16T17:14:32.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj70 = obj75 ']' 2024-09-16T17:14:32.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj71 = obj75 ']' 2024-09-16T17:14:32.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj72 = obj75 ']' 2024-09-16T17:14:32.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj73 = obj75 ']' 2024-09-16T17:14:32.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj74 = obj75 ']' 2024-09-16T17:14:32.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj75 = obj75 ']' 2024-09-16T17:14:32.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:406: TEST_rep_recovery_unfound: rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.314 INFO:tasks.workunit.client.0.smithi039.stderr:error getting test-pool/obj75: (2) No such file or directory 2024-09-16T17:14:32.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj76 = obj75 ']' 2024-09-16T17:14:32.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj77 = obj75 ']' 2024-09-16T17:14:32.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj78 = obj75 ']' 2024-09-16T17:14:32.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj79 = obj75 ']' 2024-09-16T17:14:32.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj80 = obj75 ']' 2024-09-16T17:14:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj81 = obj75 ']' 2024-09-16T17:14:32.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj82 = obj75 ']' 2024-09-16T17:14:32.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj83 = obj75 ']' 2024-09-16T17:14:32.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj84 = obj75 ']' 2024-09-16T17:14:32.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj85 = obj75 ']' 2024-09-16T17:14:32.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj86 = obj75 ']' 2024-09-16T17:14:32.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj87 = obj75 ']' 2024-09-16T17:14:32.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj88 = obj75 ']' 2024-09-16T17:14:32.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj89 = obj75 ']' 2024-09-16T17:14:32.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj90 = obj75 ']' 2024-09-16T17:14:32.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj91 = obj75 ']' 2024-09-16T17:14:32.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj92 = obj75 ']' 2024-09-16T17:14:32.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj93 = obj75 ']' 2024-09-16T17:14:32.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj94 = obj75 ']' 2024-09-16T17:14:32.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj95 = obj75 ']' 2024-09-16T17:14:32.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj96 = obj75 ']' 2024-09-16T17:14:32.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj97 = obj75 ']' 2024-09-16T17:14:32.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:32.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:32.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj98 = obj75 ']' 2024-09-16T17:14:32.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:33.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:33.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:33.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj99 = obj75 ']' 2024-09-16T17:14:33.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:33.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:33.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T17:14:33.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj100 = obj75 ']' 2024-09-16T17:14:33.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:33.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:33.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:413: TEST_rep_recovery_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-16T17:14:33.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:415: TEST_rep_recovery_unfound: delete_pool test-pool 2024-09-16T17:14:33.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-09-16T17:14:33.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-09-16T17:14:33.756 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test-pool' does not exist 2024-09-16T17:14:33.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-16T17:14:33.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:14:33.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:14:33.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-16T17:14:33.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:14:33.778 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:14:33.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:14:33.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:14:33.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:14:34.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:14:34.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:14:34.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:14:34.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:14:34.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:14:34.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:14:34.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:14:34.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:14:34.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:14:34.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:14:34.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:14:34.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:14:34.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:14:34.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-16T17:14:34.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:14:34.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:14:34.131 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:14:34.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383662 2024-09-16T17:14:34.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:14:34.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:14:34.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T17:14:34.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-rep-recov-eio.sh 0 2024-09-16T17:14:34.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-16T17:14:34.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T17:14:34.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-16T17:14:34.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:14:34.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:14:34.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:14:34.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:14:34.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:14:34.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:14:34.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:14:34.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:14:34.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:14:34.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:14:34.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:14:34.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:14:34.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:14:34.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:14:34.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:14:34.141 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:14:34.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:14:34.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T17:14:34.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-16T17:14:34.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:14:34.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:14:34.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383662 2024-09-16T17:14:34.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383662 2024-09-16T17:14:34.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:14:34.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:14:34.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T17:14:34.148 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T17:14:34.148 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T17:14:34.203 INFO:tasks.workunit:Running workunit osd/osd-reuse-id.sh... 2024-09-16T17:14:34.203 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-reuse-id.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh 2024-09-16T17:14:34.259 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T17:14:34.265 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T17:14:34.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T17:14:34.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T17:14:34.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T17:14:34.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T17:14:34.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T17:14:34.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T17:14:34.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T17:14:34.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T17:14:34.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-reuse-id 2024-09-16T17:14:34.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:20: run: local dir=td/osd-reuse-id 2024-09-16T17:14:34.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:21: run: shift 2024-09-16T17:14:34.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: export CEPH_MON=127.0.0.1:7123 2024-09-16T17:14:34.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: CEPH_MON=127.0.0.1:7123 2024-09-16T17:14:34.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:24: run: export CEPH_ARGS 2024-09-16T17:14:34.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: uuidgen 2024-09-16T17:14:34.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: CEPH_ARGS+='--fsid=4956563c-9dac-4899-89c2-cbeccdfbaf64 --auth-supported=none ' 2024-09-16T17:14:34.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7123 ' 2024-09-16T17:14:34.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: set 2024-09-16T17:14:34.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T17:14:34.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: local funcs=TEST_reuse_id 2024-09-16T17:14:34.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:29: run: for func in $funcs 2024-09-16T17:14:34.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:30: run: setup td/osd-reuse-id 2024-09-16T17:14:34.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reuse-id 2024-09-16T17:14:34.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reuse-id 2024-09-16T17:14:34.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-09-16T17:14:34.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:14:34.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-09-16T17:14:34.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:14:34.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:14:34.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:14:34.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:14:34.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:14:34.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:14:34.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:14:34.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:14:34.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:14:34.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:14:34.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:14:34.275 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:14:34.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:14:34.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:14:34.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:14:34.276 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:14:34.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:14:34.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:14:34.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-09-16T17:14:34.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:14:34.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:14:34.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:14:34.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.439376 2024-09-16T17:14:34.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:14:34.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:14:34.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reuse-id 2024-09-16T17:14:34.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:14:34.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:14:34.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:14:34.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.439376 2024-09-16T17:14:34.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:14:34.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reuse-id 1' TERM HUP INT 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:31: run: TEST_reuse_id td/osd-reuse-id 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:37: TEST_reuse_id: local dir=td/osd-reuse-id 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:39: TEST_reuse_id: run_mon td/osd-reuse-id a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reuse-id 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reuse-id/a 2024-09-16T17:14:34.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reuse-id/a --run-dir=td/osd-reuse-id --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T17:14:34.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:14:34.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:14:34.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:14:34.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:14:34.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:14:34.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:14:34.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.439376/$cluster-$name.asok' 2024-09-16T17:14:34.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reuse-id/a '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.439376/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reuse-id/log --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T17:14:34.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:14:34.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:14:34.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:14:34.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:14:34.927 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:14:34.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:14:34.929 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:14:34.929 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:14:34.929 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:14:34.929 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:14:34.929 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:14:34.929 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:14:34.929 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.439376/ceph-mon.a.asok 2024-09-16T17:14:34.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:14:34.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.439376/ceph-mon.a.asok config get fsid 2024-09-16T17:14:35.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:14:35.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:14:35.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:14:35.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:14:35.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:14:35.046 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:14:35.046 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:14:35.046 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:14:35.046 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:14:35.046 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:14:35.046 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:14:35.047 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.439376/ceph-mon.a.asok 2024-09-16T17:14:35.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:14:35.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.439376/ceph-mon.a.asok config get mon_host 2024-09-16T17:14:35.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:40: TEST_reuse_id: run_mgr td/osd-reuse-id x 2024-09-16T17:14:35.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reuse-id 2024-09-16T17:14:35.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:14:35.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:14:35.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:14:35.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reuse-id/x 2024-09-16T17:14:35.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:14:36.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:14:36.073 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:14:36.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:14:36.074 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:14:36.074 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:14:36.074 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:14:36.074 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.439376/$cluster-$name.asok' 2024-09-16T17:14:36.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:14:36.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reuse-id/x '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.439376/$cluster-$name.asok' --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:14:36.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:41: TEST_reuse_id: run_osd td/osd-reuse-id 0 2024-09-16T17:14:36.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-09-16T17:14:36.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:14:36.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:14:36.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:14:36.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/0 2024-09-16T17:14:36.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4956563c-9dac-4899-89c2-cbeccdfbaf64 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-09-16T17:14:36.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/0' 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/0/journal' 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:14:36.154 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:14:36.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.439376/$cluster-$name.asok' 2024-09-16T17:14:36.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.439376/$cluster-$name.asok' 2024-09-16T17:14:36.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/0 2024-09-16T17:14:36.156 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:14:36.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ba49d79-57e2-4878-abcd-a21bbb1869ce 2024-09-16T17:14:36.157 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 7ba49d79-57e2-4878-abcd-a21bbb1869ce 2024-09-16T17:14:36.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7ba49d79-57e2-4878-abcd-a21bbb1869ce' 2024-09-16T17:14:36.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:14:36.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8Z+hmAf+CChAAAutgc8HgFBVsxUSjcmgmug== 2024-09-16T17:14:36.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8Z+hmAf+CChAAAutgc8HgFBVsxUSjcmgmug=="}' 2024-09-16T17:14:36.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ba49d79-57e2-4878-abcd-a21bbb1869ce -i td/osd-reuse-id/0/new.json 2024-09-16T17:14:36.524 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:14:36.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/0/new.json 2024-09-16T17:14:36.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4956563c-9dac-4899-89c2-cbeccdfbaf64 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.439376/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8Z+hmAf+CChAAAutgc8HgFBVsxUSjcmgmug== --osd-uuid 7ba49d79-57e2-4878-abcd-a21bbb1869ce 2024-09-16T17:14:36.573 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:36.570+0000 7f10b3daf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:14:36.610 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:36.606+0000 7f10b3daf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:14:36.612 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:36.610+0000 7f10b3daf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:14:36.613 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:36.610+0000 7f10b3daf7c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-09-16T17:14:36.613 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:36.610+0000 7f10b3daf7c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-09-16T17:14:36.613 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:36.610+0000 7f10b3daf7c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-09-16T17:14:36.660 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:36.658+0000 7f10b3daf7c0 -1 bluestore(td/osd-reuse-id/0) _read_fsid unparsable uuid 2024-09-16T17:14:43.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/0/keyring 2024-09-16T17:14:43.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:14:43.115 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:14:43.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:14:43.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:14:43.683 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:14:43.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:14:43.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:14:43.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:14:43.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4956563c-9dac-4899-89c2-cbeccdfbaf64 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.439376/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:14:43.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:14:43.715 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:43.714+0000 7f15a2a6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:14:43.717 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:43.714+0000 7f15a2a6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:14:43.720 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:43.718+0000 7f15a2a6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:14:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:14:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:14:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:14:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:14:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:14:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:14:44.129 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:14:44.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:44.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:44.280 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:44.278+0000 7f15a2a6b7c0 -1 Falling back to public interface 2024-09-16T17:14:44.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:14:45.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:14:45.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:45.565 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:14:45.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:14:45.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:45.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:45.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:14:46.883 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:14:46.882+0000 7f15a2a6b7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:14:46.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:14:46.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:46.995 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:14:46.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:14:46.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:46.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:47.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:14:48.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:14:48.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:48.516 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:14:48.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:14:48.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:48.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:48.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:14:49.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:14:49.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:49.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:14:49.967 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:14:49.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:49.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:50.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:14:51.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:14:51.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:51.408 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:14:51.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:14:51.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:51.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:51.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:14:52.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:14:52.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:52.966 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:14:52.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:14:52.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:52.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:53.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:14:54.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:14:54.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:54.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:14:54.402 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:14:54.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:54.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:54.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:14:55.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:14:55.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:55.966 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:14:55.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:14:55.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:55.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:56.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:14:57.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:14:57.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:57.395 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:14:57.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:14:57.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:57.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:57.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:14:58.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:14:58.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:14:58.943 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:14:58.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:14:58.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:14:58.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:14:59.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:00.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:00.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:00.382 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:15:00.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:15:00.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:00.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:15:01.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:02.180 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:15:02.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:15:02.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:02.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:15:03.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:04.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:04.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:04.333 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:15:04.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:15:04.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:04.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:15:05.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:06.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:06.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:06.632 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:15:06.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:15:06.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:06.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:15:07.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:08.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:08.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:08.626 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:15:08.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:15:08.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:08.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:15:09.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:10.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:10.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:10.394 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:15:10.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:15:10.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:10.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:15:10.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:11.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:11.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:11.974 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:15:11.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:15:11.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:11.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:15:12.386 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/748885427,v1:127.0.0.1:6803/748885427] [v2:127.0.0.1:6804/748885427,v1:127.0.0.1:6805/748885427] exists,up 7ba49d79-57e2-4878-abcd-a21bbb1869ce 2024-09-16T17:15:12.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:15:12.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:15:12.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:15:12.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:42: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-09-16T17:15:12.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-09-16T17:15:12.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:15:12.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T17:15:12.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:15:12.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-09-16T17:15:12.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4956563c-9dac-4899-89c2-cbeccdfbaf64 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-09-16T17:15:12.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.439376/$cluster-$name.asok' 2024-09-16T17:15:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.439376/$cluster-$name.asok' 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:15:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-09-16T17:15:12.392 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:15:12.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1cd73d71-da98-46c7-97b6-0f2f847fe775 2024-09-16T17:15:12.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1cd73d71-da98-46c7-97b6-0f2f847fe775' 2024-09-16T17:15:12.392 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 1cd73d71-da98-46c7-97b6-0f2f847fe775 2024-09-16T17:15:12.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:15:12.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgZ+hm2+g2GBAA8mC6PR+s/ppiuCVISojv7g== 2024-09-16T17:15:12.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgZ+hm2+g2GBAA8mC6PR+s/ppiuCVISojv7g=="}' 2024-09-16T17:15:12.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1cd73d71-da98-46c7-97b6-0f2f847fe775 -i td/osd-reuse-id/1/new.json 2024-09-16T17:15:13.014 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:15:13.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-09-16T17:15:13.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4956563c-9dac-4899-89c2-cbeccdfbaf64 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.439376/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgZ+hm2+g2GBAA8mC6PR+s/ppiuCVISojv7g== --osd-uuid 1cd73d71-da98-46c7-97b6-0f2f847fe775 2024-09-16T17:15:13.068 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:13.065+0000 7f4fe07437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:15:13.081 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:13.077+0000 7f4fe07437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:15:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:13.081+0000 7f4fe07437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:15:13.083 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:13.081+0000 7f4fe07437c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-16T17:15:13.083 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:13.081+0000 7f4fe07437c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-16T17:15:13.083 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:13.081+0000 7f4fe07437c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-16T17:15:13.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:13.177+0000 7f4fe07437c0 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-09-16T17:15:19.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-09-16T17:15:19.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:15:19.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T17:15:19.260 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T17:15:19.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:15:19.827 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:15:19.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T17:15:19.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:15:19.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4956563c-9dac-4899-89c2-cbeccdfbaf64 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.439376/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:15:19.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:15:19.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:15:19.858 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:19.857+0000 7f2bb063a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:15:19.861 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:19.857+0000 7f2bb063a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:15:19.863 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:19.861+0000 7f2bb063a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:15:20.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T17:15:20.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:15:20.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:15:20.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:15:20.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:15:20.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:20.293 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:15:20.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:15:20.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:20.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:20.452 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:20.449+0000 7f2bb063a7c0 -1 Falling back to public interface 2024-09-16T17:15:20.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:21.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:21.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:21.714 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:15:21.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:15:21.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:21.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:22.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:23.116 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:23.113+0000 7f2bb063a7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T17:15:23.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:23.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:23.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:15:23.136 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:15:23.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:23.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:23.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:24.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:24.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:24.671 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:15:24.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:15:24.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:24.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:25.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:26.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:26.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:26.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:15:26.246 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:15:26.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:26.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:26.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:27.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:27.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:27.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:15:27.674 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:15:27.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:27.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:28.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:29.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:29.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:29.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:15:29.149 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:15:29.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:29.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:29.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:30.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:30.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:30.687 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:15:30.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:15:30.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:30.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:31.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:32.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:32.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:32.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:15:32.186 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:15:32.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:32.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:32.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:33.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:33.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:33.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:15:33.634 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:15:33.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:33.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:34.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:35.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:35.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:35.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:15:35.120 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:15:35.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:35.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:35.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:36.576 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:15:36.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:15:36.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:36.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:37.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:38.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:38.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:38.061 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:15:38.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:15:38.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:38.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:39.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:40.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:40.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:40.067 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:15:40.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:15:40.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:40.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:40.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:41.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:41.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:41.831 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:15:41.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:15:41.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:41.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:42.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:43.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:43.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:43.688 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:15:43.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:15:43.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:43.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:44.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:45.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:45.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:45.923 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:15:45.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:15:45.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:45.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:46.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:15:47.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:15:47.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:15:47.720 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:15:47.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:15:47.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:15:47.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:15:48.159 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1708541917,v1:127.0.0.1:6811/1708541917] [v2:127.0.0.1:6812/1708541917,v1:127.0.0.1:6813/1708541917] exists,up 1cd73d71-da98-46c7-97b6-0f2f847fe775 2024-09-16T17:15:48.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:15:48.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:15:48.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:15:48.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:43: TEST_reuse_id: create_rbd_pool 2024-09-16T17:15:48.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T17:15:48.601 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' does not exist 2024-09-16T17:15:48.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T17:15:48.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T17:15:49.346 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'rbd' already exists 2024-09-16T17:15:49.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T17:15:50.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T17:15:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:44: TEST_reuse_id: wait_for_clean 2024-09-16T17:15:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:15:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:15:52.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:15:52.208 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:15:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:15:52.208 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:15:52.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:15:52.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:15:52.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:15:52.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:15:52.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:15:52.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:15:52.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:15:52.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:15:52.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:15:52.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:15:52.758 INFO:tasks.workunit.client.0.smithi039.stderr:1' 2024-09-16T17:15:52.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:15:52.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:15:52.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:15:52.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836499 2024-09-16T17:15:52.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836499 2024-09-16T17:15:52.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836499' 2024-09-16T17:15:52.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:15:52.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:15:53.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672965 2024-09-16T17:15:53.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672965 2024-09-16T17:15:53.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836499 1-42949672965' 2024-09-16T17:15:53.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:15:53.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836499 2024-09-16T17:15:53.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:15:53.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:15:53.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836499 2024-09-16T17:15:53.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:15:53.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836499 2024-09-16T17:15:53.116 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836499 2024-09-16T17:15:53.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-16T17:15:53.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:15:53.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836497 -lt 21474836499 2024-09-16T17:15:53.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:15:54.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:15:54.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:15:55.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-16T17:15:55.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:15:55.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672965 2024-09-16T17:15:55.037 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:15:55.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:15:55.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672965 2024-09-16T17:15:55.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:15:55.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672965 2024-09-16T17:15:55.040 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672965 2024-09-16T17:15:55.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-16T17:15:55.041 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:15:55.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672966 -lt 42949672965 2024-09-16T17:15:55.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:15:55.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:15:55.503 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:15:56.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T17:15:56.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:15:56.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:15:56.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:15:56.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:15:56.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:15:56.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:15:56.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:15:56.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T17:15:56.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:15:56.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:15:56.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:15:57.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T17:15:57.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:15:57.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:15:57.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:45: TEST_reuse_id: destroy_osd td/osd-reuse-id 1 2024-09-16T17:15:57.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/osd-reuse-id 2024-09-16T17:15:57.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=1 2024-09-16T17:15:57.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.1 2024-09-16T17:15:57.737 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already out. 2024-09-16T17:15:57.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/osd-reuse-id TERM osd.1 2024-09-16T17:15:57.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:15:57.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:15:57.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:15:57.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:15:57.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:15:58.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:15:58.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.1 2024-09-16T17:15:58.492 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-16T17:15:58.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.1 --yes-i-really-mean-it 2024-09-16T17:15:59.088 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 does not exist 2024-09-16T17:15:59.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/osd-reuse-id/1 2024-09-16T17:15:59.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id/1 2024-09-16T17:15:59.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:15:59.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id/1 KILL 2024-09-16T17:15:59.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:15:59.112 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:15:59.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:15:59.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:15:59.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:15:59.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:15:59.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:15:59.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:15:59.118 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:15:59.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:15:59.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:15:59.120 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:15:59.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:15:59.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:15:59.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:15:59.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:15:59.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:15:59.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:15:59.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id/1 2024-09-16T17:15:59.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:15:59.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:15:59.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:15:59.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.439376 2024-09-16T17:15:59.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:15:59.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:15:59.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/osd-reuse-id/1 2024-09-16T17:15:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:46: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-09-16T17:15:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-09-16T17:15:59.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4956563c-9dac-4899-89c2-cbeccdfbaf64 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:15:59.133 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:15:59.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:15:59.134 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:15:59.134 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:15:59.134 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:15:59.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.439376/$cluster-$name.asok' 2024-09-16T17:15:59.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.439376/$cluster-$name.asok' 2024-09-16T17:15:59.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:15:59.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:15:59.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:15:59.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-09-16T17:15:59.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-09-16T17:15:59.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:15:59.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:15:59.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:15:59.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:15:59.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:15:59.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:15:59.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-09-16T17:15:59.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:15:59.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e26bd733-68c1-472f-861e-3ea65eaffac5 2024-09-16T17:15:59.137 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 e26bd733-68c1-472f-861e-3ea65eaffac5 2024-09-16T17:15:59.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e26bd733-68c1-472f-861e-3ea65eaffac5' 2024-09-16T17:15:59.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:15:59.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPZ+hmVMBKCRAA0pF7RWjd69Tye6+U5l4aFg== 2024-09-16T17:15:59.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPZ+hmVMBKCRAA0pF7RWjd69Tye6+U5l4aFg=="}' 2024-09-16T17:15:59.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e26bd733-68c1-472f-861e-3ea65eaffac5 -i td/osd-reuse-id/1/new.json 2024-09-16T17:15:59.752 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:15:59.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-09-16T17:15:59.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4956563c-9dac-4899-89c2-cbeccdfbaf64 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.439376/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPZ+hmVMBKCRAA0pF7RWjd69Tye6+U5l4aFg== --osd-uuid e26bd733-68c1-472f-861e-3ea65eaffac5 2024-09-16T17:15:59.806 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:59.800+0000 7fe6346007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:15:59.850 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:59.844+0000 7fe6346007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:15:59.852 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:59.848+0000 7fe6346007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:15:59.852 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:59.848+0000 7fe6346007c0 -1 asok(0x55fd42f1a000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.439376/ceph-osd.1.asok': (2) No such file or directory 2024-09-16T17:15:59.852 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:59.848+0000 7fe6346007c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-16T17:15:59.853 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:59.848+0000 7fe6346007c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-16T17:15:59.853 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:59.848+0000 7fe6346007c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-16T17:15:59.950 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:15:59.944+0000 7fe6346007c0 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-09-16T17:16:06.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-09-16T17:16:06.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:16:06.202 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T17:16:06.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T17:16:06.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:16:06.817 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:16:06.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T17:16:06.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4956563c-9dac-4899-89c2-cbeccdfbaf64 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.439376/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:16:06.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:16:06.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:16:06.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:16:06.847 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:06.844+0000 7fd6e5aca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:06.850 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:06.844+0000 7fd6e5aca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:06.851 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:06.848+0000 7fd6e5aca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:06.852 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:06.848+0000 7fd6e5aca7c0 -1 asok(0x561b49750000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.439376/ceph-osd.1.asok': (2) No such file or directory 2024-09-16T17:16:07.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T17:16:07.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:16:07.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:16:07.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:16:07.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:16:07.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:07.338 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:16:07.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:16:07.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:07.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:16:07.429 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:07.424+0000 7fd6e5aca7c0 -1 Falling back to public interface 2024-09-16T17:16:07.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:08.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:08.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:08.782 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:16:08.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:16:08.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:08.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:16:09.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:09.965 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:09.960+0000 7fd6e5aca7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T17:16:10.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:10.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:10.230 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:16:10.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:16:10.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:10.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:16:10.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:11.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:11.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:11.756 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:16:11.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:16:11.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:11.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:16:12.179 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3750713223,v1:127.0.0.1:6811/3750713223] [v2:127.0.0.1:6812/3750713223,v1:127.0.0.1:6813/3750713223] exists,up e26bd733-68c1-472f-861e-3ea65eaffac5 2024-09-16T17:16:12.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:16:12.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:16:12.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:16:12.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:32: run: teardown td/osd-reuse-id 2024-09-16T17:16:12.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-09-16T17:16:12.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:16:12.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-09-16T17:16:12.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:16:12.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:16:12.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:16:12.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:16:12.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:16:12.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:16:12.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:16:12.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:16:12.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:16:12.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:16:12.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:16:12.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:16:12.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:16:12.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:16:12.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:16:12.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:16:12.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:16:12.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:16:12.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-09-16T17:16:12.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:16:12.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:16:12.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:16:12.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.439376 2024-09-16T17:16:12.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:16:12.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:16:12.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T17:16:12.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-reuse-id 0 2024-09-16T17:16:12.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-09-16T17:16:12.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T17:16:12.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-09-16T17:16:12.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:16:12.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:16:12.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:16:12.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:16:12.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:16:12.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:16:12.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:16:12.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:16:12.514 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:16:12.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:16:12.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:16:12.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:16:12.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:16:12.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:16:12.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:16:12.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:16:12.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:16:12.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T17:16:12.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-09-16T17:16:12.520 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:16:12.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:16:12.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.439376 2024-09-16T17:16:12.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.439376 2024-09-16T17:16:12.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:16:12.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:16:12.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T17:16:12.523 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T17:16:12.523 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T17:16:12.579 INFO:tasks.workunit:Running workunit osd/pg-split-merge.sh... 2024-09-16T17:16:12.579 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/pg-split-merge.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh 2024-09-16T17:16:12.634 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T17:16:12.640 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T17:16:12.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T17:16:12.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T17:16:12.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T17:16:12.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T17:16:12.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T17:16:12.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T17:16:12.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T17:16:12.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T17:16:12.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/pg-split-merge 2024-09-16T17:16:12.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:5: run: local dir=td/pg-split-merge 2024-09-16T17:16:12.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:6: run: shift 2024-09-16T17:16:12.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: export CEPH_MON=127.0.0.1:7147 2024-09-16T17:16:12.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: CEPH_MON=127.0.0.1:7147 2024-09-16T17:16:12.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:9: run: export CEPH_ARGS 2024-09-16T17:16:12.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: uuidgen 2024-09-16T17:16:12.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: CEPH_ARGS+='--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none ' 2024-09-16T17:16:12.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:16:12.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: set 2024-09-16T17:16:12.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T17:16:12.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: local 'funcs=TEST_a_merge_empty 2024-09-16T17:16:12.644 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_import_after_merge_and_gap 2024-09-16T17:16:12.644 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_import_after_split' 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:16:12.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:16:12.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:16:12.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:16:12.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:16:12.648 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:16:12.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:16:12.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:16:12.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:16:12.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:16:12.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:16:12.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:16:12.651 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:16:12.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:16:12.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:16:12.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-16T17:16:12.654 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:16:12.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:16:12.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:16:12.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.444977 2024-09-16T17:16:12.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:16:12.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:16:12.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-09-16T17:16:12.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:16:12.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:16:12.658 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:16:12.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.444977 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_a_merge_empty td/pg-split-merge 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:22: TEST_a_merge_empty: local dir=td/pg-split-merge 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:24: TEST_a_merge_empty: run_mon td/pg-split-merge a --osd_pool_default_size=3 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:16:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:16:12.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-09-16T17:16:12.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=3 2024-09-16T17:16:13.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:16:13.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:16:13.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:16:13.040 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:16:13.040 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:16:13.040 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:16:13.040 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:16:13.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-16T17:16:13.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:16:13.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:16:13.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:16:13.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:16:13.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:16:13.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:16:13.288 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:16:13.288 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:16:13.288 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:16:13.288 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:16:13.288 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:16:13.288 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:16:13.289 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.444977/ceph-mon.a.asok 2024-09-16T17:16:13.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:16:13.289 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.444977/ceph-mon.a.asok config get fsid 2024-09-16T17:16:13.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:16:13.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:16:13.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:16:13.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:16:13.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:16:13.408 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:16:13.408 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:16:13.408 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:16:13.408 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:16:13.408 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:16:13.408 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:16:13.408 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.444977/ceph-mon.a.asok 2024-09-16T17:16:13.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:16:13.408 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.444977/ceph-mon.a.asok config get mon_host 2024-09-16T17:16:13.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:25: TEST_a_merge_empty: run_mgr td/pg-split-merge x 2024-09-16T17:16:13.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-09-16T17:16:13.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:16:13.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:16:13.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:16:13.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-09-16T17:16:13.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:16:14.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:16:14.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:16:14.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:16:14.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:16:14.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:16:14.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:16:14.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:16:14.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:16:14.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:16:14.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:26: TEST_a_merge_empty: run_osd td/pg-split-merge 0 2024-09-16T17:16:14.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-16T17:16:14.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:16:14.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:16:14.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:16:14.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-09-16T17:16:14.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:16:14.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:16:14.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:16:14.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:16:14.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:16:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-09-16T17:16:14.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:16:14.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=65c3ba20-f0e3-4317-b614-b35252156ce9 2024-09-16T17:16:14.512 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 65c3ba20-f0e3-4317-b614-b35252156ce9 2024-09-16T17:16:14.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 65c3ba20-f0e3-4317-b614-b35252156ce9' 2024-09-16T17:16:14.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:16:14.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDeZ+hmBoeVHxAAzpGQuQUSBxDkZzdIFtMk8g== 2024-09-16T17:16:14.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDeZ+hmBoeVHxAAzpGQuQUSBxDkZzdIFtMk8g=="}' 2024-09-16T17:16:14.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 65c3ba20-f0e3-4317-b614-b35252156ce9 -i td/pg-split-merge/0/new.json 2024-09-16T17:16:14.891 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:16:14.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-09-16T17:16:14.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDeZ+hmBoeVHxAAzpGQuQUSBxDkZzdIFtMk8g== --osd-uuid 65c3ba20-f0e3-4317-b614-b35252156ce9 2024-09-16T17:16:14.943 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:14.940+0000 7f963c5977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:14.945 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:14.940+0000 7f963c5977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:14.947 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:14.944+0000 7f963c5977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:14.947 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:14.944+0000 7f963c5977c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-16T17:16:14.947 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:14.944+0000 7f963c5977c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-16T17:16:14.947 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:14.944+0000 7f963c5977c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-16T17:16:15.022 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:15.016+0000 7f963c5977c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-09-16T17:16:21.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-09-16T17:16:21.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:16:21.178 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:16:21.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:16:21.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:16:21.731 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:16:21.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:16:21.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:16:21.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:16:21.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:16:21.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:16:21.767 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:21.764+0000 7f8ffa5c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:21.769 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:21.764+0000 7f8ffa5c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:21.770 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:21.768+0000 7f8ffa5c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:22.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:16:22.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:16:22.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:16:22.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:16:22.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:16:22.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:22.197 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:16:22.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:16:22.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:22.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:22.332 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:22.328+0000 7f8ffa5c57c0 -1 Falling back to public interface 2024-09-16T17:16:22.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:23.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:23.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:23.672 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:16:23.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:16:23.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:23.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:24.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:24.950 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:24.944+0000 7f8ffa5c57c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:16:25.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:25.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:25.093 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:16:25.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:16:25.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:25.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:25.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:26.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:26.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:26.664 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:16:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:16:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:26.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:27.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:28.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:28.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:28.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:16:28.108 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:16:28.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:28.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:28.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:29.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:29.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:29.650 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:16:29.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:16:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:29.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:30.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:31.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:31.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:31.081 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:16:31.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:16:31.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:31.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:31.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:32.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:32.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:32.670 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:16:32.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:16:32.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:32.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:33.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:34.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:34.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:34.105 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:16:34.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:16:34.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:34.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:34.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:35.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:35.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:35.665 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:16:35.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:16:35.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:35.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:36.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:37.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:37.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:37.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:16:37.158 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:16:37.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:37.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:37.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:38.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:38.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:38.743 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:16:38.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:16:38.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:38.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:39.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:40.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:40.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:40.259 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:16:40.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:16:40.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:40.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:41.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:42.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:42.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:42.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:16:42.081 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:16:42.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:42.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:43.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:44.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:44.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:44.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:16:44.196 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:16:44.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:44.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:45.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:46.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:46.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:46.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:16:46.042 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:16:46.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:46.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:46.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:47.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:47.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:47.997 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:16:47.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:16:47.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:47.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:49.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:16:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:16:50.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:50.012 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:16:50.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:16:50.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:50.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:16:50.469 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2994810201,v1:127.0.0.1:6803/2994810201] [v2:127.0.0.1:6804/2994810201,v1:127.0.0.1:6805/2994810201] exists,up 65c3ba20-f0e3-4317-b614-b35252156ce9 2024-09-16T17:16:50.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:16:50.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:16:50.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:16:50.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:27: TEST_a_merge_empty: run_osd td/pg-split-merge 1 2024-09-16T17:16:50.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-16T17:16:50.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:16:50.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T17:16:50.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:16:50.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/1 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/1' 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/1/journal' 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:16:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:16:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/1 2024-09-16T17:16:50.475 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:16:50.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=35782b3a-bcdb-4f25-aff4-16a92ef44114 2024-09-16T17:16:50.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 35782b3a-bcdb-4f25-aff4-16a92ef44114' 2024-09-16T17:16:50.475 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 35782b3a-bcdb-4f25-aff4-16a92ef44114 2024-09-16T17:16:50.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:16:50.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACaOhms99FHRAA8ciJkAARVNqKJgFOSIkrlw== 2024-09-16T17:16:50.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACaOhms99FHRAA8ciJkAARVNqKJgFOSIkrlw=="}' 2024-09-16T17:16:50.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 35782b3a-bcdb-4f25-aff4-16a92ef44114 -i td/pg-split-merge/1/new.json 2024-09-16T17:16:51.059 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:16:51.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/1/new.json 2024-09-16T17:16:51.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACaOhms99FHRAA8ciJkAARVNqKJgFOSIkrlw== --osd-uuid 35782b3a-bcdb-4f25-aff4-16a92ef44114 2024-09-16T17:16:51.118 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:51.111+0000 7f60774e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:51.120 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:51.115+0000 7f60774e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:51.121 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:51.115+0000 7f60774e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:51.122 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:51.115+0000 7f60774e57c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-09-16T17:16:51.122 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:51.115+0000 7f60774e57c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-09-16T17:16:51.122 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:51.115+0000 7f60774e57c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-09-16T17:16:51.226 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:51.219+0000 7f60774e57c0 -1 bluestore(td/pg-split-merge/1) _read_fsid unparsable uuid 2024-09-16T17:16:57.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/1/keyring 2024-09-16T17:16:57.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:16:57.868 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T17:16:57.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T17:16:57.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:16:58.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T17:16:58.432 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:16:58.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:16:58.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:16:58.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:16:58.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:16:58.458 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:58.451+0000 7f6ab60c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:58.460 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:58.455+0000 7f6ab60c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:58.462 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:58.455+0000 7f6ab60c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:16:58.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T17:16:58.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:16:58.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:16:58.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:16:58.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:16:58.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:16:58.878 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:16:58.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:16:58.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:16:58.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:16:59.019 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:16:59.015+0000 7f6ab60c67c0 -1 Falling back to public interface 2024-09-16T17:16:59.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:00.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:00.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:00.320 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:17:00.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:17:00.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:00.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:00.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:01.746 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:01.739+0000 7f6ab60c67c0 -1 osd.1 0 log_to_monitors true 2024-09-16T17:17:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:01.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:17:01.757 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:17:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:01.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:02.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:03.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:03.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:03.195 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:17:03.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:17:03.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:03.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:03.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:04.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:04.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:04.739 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:17:04.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:17:04.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:04.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:05.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:06.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:06.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:06.179 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:17:06.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:17:06.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:06.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:06.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:07.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:07.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:07.721 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:17:07.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:17:07.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:07.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:08.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:09.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:09.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:09.211 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:17:09.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:17:09.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:09.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:09.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:10.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:10.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:10.838 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:17:10.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:17:10.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:10.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:11.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:12.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:12.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:12.278 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:17:12.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:17:12.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:12.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:12.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:13.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:13.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:13.781 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:17:13.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:17:13.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:13.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:14.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:15.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:15.342 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:17:15.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:15.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:17:15.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:15.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:15.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:16.937 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:17:16.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:16.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:16.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:17:16.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:16.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:17.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:18.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:18.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:18.613 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:17:18.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:17:18.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:18.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:19.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:20.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:20.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:20.480 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:17:20.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:17:20.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:20.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:21.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:22.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:22.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:22.737 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:17:22.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:17:22.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:22.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:23.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:24.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:24.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:24.631 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:17:24.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:17:24.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:24.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:25.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:26.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:26.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:26.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:17:26.811 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:17:26.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:26.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:27.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:28.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:28.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:28.632 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:17:28.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T17:17:28.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:28.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:17:29.083 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1674428305,v1:127.0.0.1:6811/1674428305] [v2:127.0.0.1:6812/1674428305,v1:127.0.0.1:6813/1674428305] exists,up 35782b3a-bcdb-4f25-aff4-16a92ef44114 2024-09-16T17:17:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:17:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:17:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:17:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:28: TEST_a_merge_empty: run_osd td/pg-split-merge 2 2024-09-16T17:17:29.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/2 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:17:29.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:17:29.087 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:17:29.087 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:17:29.087 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:17:29.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:17:29.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:17:29.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:17:29.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:17:29.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:17:29.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-16T17:17:29.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-16T17:17:29.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:17:29.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:17:29.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:17:29.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:17:29.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:17:29.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:17:29.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/2 2024-09-16T17:17:29.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:17:29.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0eb30054-914e-4e80-99bc-7e778eb690e8 2024-09-16T17:17:29.090 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 0eb30054-914e-4e80-99bc-7e778eb690e8 2024-09-16T17:17:29.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0eb30054-914e-4e80-99bc-7e778eb690e8' 2024-09-16T17:17:29.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:17:29.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApaOhm8RxjBhAAgpJz8SfcYTEZWa1sPJDk/Q== 2024-09-16T17:17:29.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApaOhm8RxjBhAAgpJz8SfcYTEZWa1sPJDk/Q=="}' 2024-09-16T17:17:29.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0eb30054-914e-4e80-99bc-7e778eb690e8 -i td/pg-split-merge/2/new.json 2024-09-16T17:17:29.660 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:17:29.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/2/new.json 2024-09-16T17:17:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApaOhm8RxjBhAAgpJz8SfcYTEZWa1sPJDk/Q== --osd-uuid 0eb30054-914e-4e80-99bc-7e778eb690e8 2024-09-16T17:17:29.717 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:29.714+0000 7f19d89f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:17:29.720 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:29.718+0000 7f19d89f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:17:29.721 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:29.718+0000 7f19d89f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:17:29.721 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:29.718+0000 7f19d89f27c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-09-16T17:17:29.721 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:29.718+0000 7f19d89f27c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-09-16T17:17:29.722 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:29.718+0000 7f19d89f27c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-09-16T17:17:29.761 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:29.758+0000 7f19d89f27c0 -1 bluestore(td/pg-split-merge/2) _read_fsid unparsable uuid 2024-09-16T17:17:36.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/2/keyring 2024-09-16T17:17:36.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:17:36.013 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T17:17:36.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T17:17:36.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:17:36.593 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T17:17:36.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T17:17:36.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:17:36.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:17:36.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:17:36.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:17:36.629 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:36.626+0000 7f44be2f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:17:36.651 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:36.650+0000 7f44be2f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:17:36.652 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:36.650+0000 7f44be2f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:17:37.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T17:17:37.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:17:37.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:17:37.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:17:37.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:17:37.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:37.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:17:37.026 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:17:37.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:37.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:37.232 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:37.230+0000 7f44be2f47c0 -1 Falling back to public interface 2024-09-16T17:17:37.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:38.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:38.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:38.470 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:17:38.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:17:38.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:38.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:38.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:39.933 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:17:39.930+0000 7f44be2f47c0 -1 osd.2 0 log_to_monitors true 2024-09-16T17:17:39.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:39.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:39.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:17:39.953 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:17:39.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:39.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:40.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:41.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:41.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:41.700 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:17:41.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:17:41.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:41.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:42.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:43.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:43.196 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:17:43.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:43.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:17:43.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:43.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:43.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:44.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:44.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:44.846 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:17:44.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:17:44.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:44.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:45.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:46.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:46.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:46.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:17:46.339 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:17:46.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:46.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:46.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:47.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:47.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:47.844 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:17:47.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:17:47.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:47.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:48.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:49.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:49.302 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:17:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:17:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:49.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:50.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:50.805 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:17:50.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:50.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:17:50.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:50.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:51.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:52.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:52.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:52.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:17:52.287 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:17:52.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:52.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:52.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:53.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:53.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:53.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:17:53.809 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:17:53.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:53.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:54.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:55.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:55.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:55.448 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:17:55.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:17:55.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:55.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:56.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:57.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:57.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:57.460 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:17:57.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:17:57.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:57.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:58.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:17:59.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:17:59.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:17:59.115 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:17:59.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:17:59.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:17:59.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:17:59.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:18:00.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:18:00.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:18:00.589 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:18:00.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:18:00.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:18:00.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:18:01.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:18:02.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:18:02.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:18:02.410 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:18:02.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:18:02.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:18:02.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:18:03.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:18:04.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:18:04.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:18:04.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:18:04.168 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:18:04.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:18:04.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:18:05.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:18:06.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:18:06.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:18:06.088 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:18:06.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T17:18:06.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:18:06.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:18:06.524 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2600654286,v1:127.0.0.1:6819/2600654286] [v2:127.0.0.1:6820/2600654286,v1:127.0.0.1:6821/2600654286] exists,up 0eb30054-914e-4e80-99bc-7e778eb690e8 2024-09-16T17:18:06.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:18:06.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:18:06.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:18:06.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:30: TEST_a_merge_empty: ceph osd pool create foo 2 2024-09-16T17:18:07.114 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T17:18:07.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:31: TEST_a_merge_empty: ceph osd pool set foo pgp_num 1 2024-09-16T17:18:09.231 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 pgp_num to 1 2024-09-16T17:18:09.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:33: TEST_a_merge_empty: wait_for_clean 2024-09-16T17:18:09.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:18:09.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:18:09.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:18:09.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:18:09.260 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:18:09.260 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:18:09.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:18:09.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:18:09.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:18:09.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:18:09.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:18:09.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:18:09.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:18:09.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:18:09.383 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:18:09.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:18:09.990 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:18:09.990 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:18:09.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:18:09.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:09.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:18:10.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836503 2024-09-16T17:18:10.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836503 2024-09-16T17:18:10.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503' 2024-09-16T17:18:10.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:10.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:18:10.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672976 2024-09-16T17:18:10.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672976 2024-09-16T17:18:10.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672976' 2024-09-16T17:18:10.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:10.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:18:10.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509450 2024-09-16T17:18:10.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509450 2024-09-16T17:18:10.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672976 2-64424509450' 2024-09-16T17:18:10.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:10.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836503 2024-09-16T17:18:10.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:10.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:18:10.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836503 2024-09-16T17:18:10.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:10.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836503 2024-09-16T17:18:10.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-16T17:18:10.530 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836503 2024-09-16T17:18:10.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:18:10.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836501 -lt 21474836503 2024-09-16T17:18:10.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:18:11.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:18:11.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:18:12.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836504 -lt 21474836503 2024-09-16T17:18:12.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:12.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672976 2024-09-16T17:18:12.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:12.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:18:12.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672976 2024-09-16T17:18:12.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:12.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672976 2024-09-16T17:18:12.390 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672976 2024-09-16T17:18:12.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-16T17:18:12.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:18:12.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-16T17:18:12.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:12.820 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509450 2024-09-16T17:18:12.821 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:12.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:18:12.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509450 2024-09-16T17:18:12.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:12.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509450 2024-09-16T17:18:12.824 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509450 2024-09-16T17:18:12.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-16T17:18:12.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:18:13.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-16T17:18:13.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:18:13.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:18:13.308 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:18:13.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-16T17:18:13.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:18:13.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:18:13.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:18:13.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:18:13.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:18:13.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:18:13.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:18:14.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T17:18:14.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:18:14.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:18:14.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:18:14.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-16T17:18:14.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:18:14.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:18:14.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:37: TEST_a_merge_empty: rados -p foo put foo1 /etc/passwd 2024-09-16T17:18:15.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:38: TEST_a_merge_empty: rados -p foo put foo2 /etc/passwd 2024-09-16T17:18:15.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:39: TEST_a_merge_empty: rados -p foo put foo3 /etc/passwd 2024-09-16T17:18:15.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:40: TEST_a_merge_empty: rados -p foo put foo4 /etc/passwd 2024-09-16T17:18:15.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:42: TEST_a_merge_empty: rados -p foo put foo5 /etc/passwd 2024-09-16T17:18:15.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:43: TEST_a_merge_empty: rados -p foo put foo6 /etc/passwd 2024-09-16T17:18:16.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:44: TEST_a_merge_empty: rados -p foo put foo8 /etc/passwd 2024-09-16T17:18:16.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:45: TEST_a_merge_empty: rados -p foo put foo10 /etc/passwd 2024-09-16T17:18:16.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:46: TEST_a_merge_empty: rados -p foo put foo11 /etc/passwd 2024-09-16T17:18:16.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:47: TEST_a_merge_empty: rados -p foo put foo12 /etc/passwd 2024-09-16T17:18:16.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:48: TEST_a_merge_empty: rados -p foo put foo16 /etc/passwd 2024-09-16T17:18:16.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:50: TEST_a_merge_empty: wait_for_clean 2024-09-16T17:18:16.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:18:16.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:18:16.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:18:16.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:18:16.864 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:18:16.864 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:18:16.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:18:16.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:18:16.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:18:16.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:18:16.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:18:16.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:18:16.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:18:16.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:18:16.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:18:17.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:18:17.428 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:18:17.428 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:18:17.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:18:17.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:17.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:18:17.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836507 2024-09-16T17:18:17.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836507 2024-09-16T17:18:17.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836507' 2024-09-16T17:18:17.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:17.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:18:17.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672979 2024-09-16T17:18:17.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672979 2024-09-16T17:18:17.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836507 1-42949672979' 2024-09-16T17:18:17.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:17.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:18:17.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509453 2024-09-16T17:18:17.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509453 2024-09-16T17:18:17.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836507 1-42949672979 2-64424509453' 2024-09-16T17:18:17.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:17.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836507 2024-09-16T17:18:17.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:17.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:18:17.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836507 2024-09-16T17:18:17.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:17.910 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836507 2024-09-16T17:18:17.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836507 2024-09-16T17:18:17.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-16T17:18:17.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:18:18.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-16T17:18:18.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:18:19.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:18:19.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:18:19.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-16T17:18:19.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:19.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672979 2024-09-16T17:18:19.814 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:19.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:18:19.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672979 2024-09-16T17:18:19.816 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672979 2024-09-16T17:18:19.817 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672979 2024-09-16T17:18:19.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-16T17:18:19.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:18:20.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672980 -lt 42949672979 2024-09-16T17:18:20.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:20.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509453 2024-09-16T17:18:20.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:20.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:18:20.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:20.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509453 2024-09-16T17:18:20.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509453 2024-09-16T17:18:20.235 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509453 2024-09-16T17:18:20.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-16T17:18:20.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:18:20.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-16T17:18:20.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:18:20.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:18:20.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:18:21.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-16T17:18:21.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:18:21.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:18:21.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:18:21.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:18:21.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:18:21.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:18:21.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:18:21.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T17:18:21.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:18:21.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:18:21.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:18:22.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-16T17:18:22.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:18:22.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:18:22.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:52: TEST_a_merge_empty: ceph tell osd.1 config set osd_debug_no_purge_strays true 2024-09-16T17:18:22.381 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T17:18:22.381 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "" 2024-09-16T17:18:22.381 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T17:18:22.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:53: TEST_a_merge_empty: ceph osd pool set foo size 2 2024-09-16T17:18:23.505 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 2 2024-09-16T17:18:23.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:54: TEST_a_merge_empty: wait_for_clean 2024-09-16T17:18:23.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:18:23.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:18:23.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:18:23.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:18:23.532 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:18:23.532 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:18:23.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:18:23.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:18:23.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:18:23.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:18:23.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:18:23.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:18:23.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:18:23.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:18:23.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:18:24.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:18:24.189 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:18:24.189 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:18:24.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:18:24.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:24.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:18:24.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836510 2024-09-16T17:18:24.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836510 2024-09-16T17:18:24.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510' 2024-09-16T17:18:24.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:24.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:18:24.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672983 2024-09-16T17:18:24.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672983 2024-09-16T17:18:24.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510 1-42949672983' 2024-09-16T17:18:24.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:24.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:18:24.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509456 2024-09-16T17:18:24.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509456 2024-09-16T17:18:24.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510 1-42949672983 2-64424509456' 2024-09-16T17:18:24.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:24.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836510 2024-09-16T17:18:24.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:24.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:18:24.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836510 2024-09-16T17:18:24.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:24.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836510 2024-09-16T17:18:24.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-16T17:18:24.730 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836510 2024-09-16T17:18:24.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:18:25.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836508 -lt 21474836510 2024-09-16T17:18:25.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:18:26.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:18:26.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:18:26.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-16T17:18:26.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:26.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672983 2024-09-16T17:18:26.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:26.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:18:26.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672983 2024-09-16T17:18:26.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:26.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672983 2024-09-16T17:18:26.569 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672983 2024-09-16T17:18:26.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-16T17:18:26.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:18:26.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-16T17:18:26.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:26.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509456 2024-09-16T17:18:26.970 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:26.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:18:26.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509456 2024-09-16T17:18:26.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:26.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509456 2024-09-16T17:18:26.973 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509456 2024-09-16T17:18:26.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-16T17:18:26.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:18:27.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509457 -lt 64424509456 2024-09-16T17:18:27.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:18:27.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:18:27.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:18:27.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-16T17:18:27.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:18:27.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:18:27.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:18:27.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:18:27.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:18:27.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:18:27.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:18:28.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T17:18:28.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:18:28.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:18:28.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:18:29.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-16T17:18:29.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:18:29.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:18:29.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:56: TEST_a_merge_empty: kill_daemons td/pg-split-merge TERM osd.2 2024-09-16T17:18:29.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:18:29.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:18:29.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:18:29.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:18:29.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:18:29.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:18:29.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:57: TEST_a_merge_empty: ceph-objectstore-tool --data-path td/pg-split-merge/2 --op remove --pgid 1.1 --force 2024-09-16T17:18:30.699 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-16T17:18:30.699 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-16T17:18:30.699 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-09-16T17:18:30.840 INFO:tasks.workunit.client.0.smithi039.stdout:Remove successful 2024-09-16T17:18:31.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:58: TEST_a_merge_empty: activate_osd td/pg-split-merge 2 2024-09-16T17:18:31.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-16T17:18:31.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T17:18:31.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T17:18:31.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T17:18:31.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/2 2024-09-16T17:18:31.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:18:31.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:18:31.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:18:31.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:18:31.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-09-16T17:18:31.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-09-16T17:18:31.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T17:18:31.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T17:18:31.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-16T17:18:31.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T17:18:31.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:18:31.234 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:18:31.235 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:18:31.235 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:18:31.235 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:18:31.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:18:31.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:18:31.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:18:31.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-16T17:18:31.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-16T17:18:31.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:18:31.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:18:31.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:18:31.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:18:31.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T17:18:31.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T17:18:31.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/2 2024-09-16T17:18:31.237 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T17:18:31.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T17:18:31.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:18:31.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/2/whoami 2024-09-16T17:18:31.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T17:18:31.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T17:18:31.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T17:18:31.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T17:18:31.273 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:18:31.269+0000 7f33d46257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:18:31.276 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:18:31.273+0000 7f33d46257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:18:31.277 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:18:31.273+0000 7f33d46257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:18:31.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T17:18:31.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:18:31.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:18:31.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:18:31.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:18:31.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:18:31.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:18:31.695 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:18:31.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:18:31.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:18:31.820 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:18:31.817+0000 7f33d46257c0 -1 Falling back to public interface 2024-09-16T17:18:32.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:18:33.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:18:33.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:18:33.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:18:33.183 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:18:33.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:18:33.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:18:33.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:18:34.481 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:18:34.477+0000 7f33d46257c0 -1 osd.2 27 log_to_monitors true 2024-09-16T17:18:34.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:18:34.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:18:34.606 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:18:34.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:18:34.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:18:34.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:18:35.036 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 30 up_thru 17 down_at 28 last_clean_interval [15,27) [v2:127.0.0.1:6818/18639260,v1:127.0.0.1:6819/18639260] [v2:127.0.0.1:6820/18639260,v1:127.0.0.1:6821/18639260] exists,up 0eb30054-914e-4e80-99bc-7e778eb690e8 2024-09-16T17:18:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:18:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:18:35.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:18:35.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:60: TEST_a_merge_empty: wait_for_clean 2024-09-16T17:18:35.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:18:35.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:18:35.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:18:35.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:18:35.039 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:18:35.039 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:18:35.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:18:35.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:18:35.039 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:18:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:18:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:18:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:18:35.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:18:35.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:18:35.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:18:35.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:18:35.558 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:18:35.559 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:18:35.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:18:35.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:35.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:18:35.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836515 2024-09-16T17:18:35.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836515 2024-09-16T17:18:35.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836515' 2024-09-16T17:18:35.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:35.731 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:18:35.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672987 2024-09-16T17:18:35.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672987 2024-09-16T17:18:35.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836515 1-42949672987' 2024-09-16T17:18:35.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:35.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:18:36.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018883 2024-09-16T17:18:36.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018883 2024-09-16T17:18:36.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836515 1-42949672987 2-128849018883' 2024-09-16T17:18:36.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:36.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836515 2024-09-16T17:18:36.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:36.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:18:36.028 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836515 2024-09-16T17:18:36.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:36.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836515 2024-09-16T17:18:36.030 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836515 2024-09-16T17:18:36.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-09-16T17:18:36.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:18:36.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836512 -lt 21474836515 2024-09-16T17:18:36.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:18:37.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:18:37.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:18:37.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836515 -lt 21474836515 2024-09-16T17:18:37.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:37.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672987 2024-09-16T17:18:37.885 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:18:37.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672987 2024-09-16T17:18:37.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:37.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672987 2024-09-16T17:18:37.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-16T17:18:37.888 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672987 2024-09-16T17:18:37.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:18:38.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-16T17:18:38.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:38.291 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-128849018883 2024-09-16T17:18:38.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:38.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:18:38.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-128849018883 2024-09-16T17:18:38.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:38.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018883 2024-09-16T17:18:38.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 128849018883' 2024-09-16T17:18:38.296 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 128849018883 2024-09-16T17:18:38.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:18:38.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018883 -lt 128849018883 2024-09-16T17:18:38.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:18:38.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:18:38.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:18:39.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-16T17:18:39.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:18:39.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:18:39.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:18:39.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:18:39.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:18:39.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:18:39.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:18:39.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T17:18:39.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:18:39.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:18:39.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:18:40.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-16T17:18:40.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:18:40.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:18:40.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:66: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change true 2024-09-16T17:18:40.497 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T17:18:40.497 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "" 2024-09-16T17:18:40.497 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T17:18:40.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:67: TEST_a_merge_empty: ceph osd out 0 1 2024-09-16T17:18:41.386 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. osd.1 is already out. 2024-09-16T17:18:41.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:68: TEST_a_merge_empty: ceph osd pool set foo pg_num 1 2024-09-16T17:18:42.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:69: TEST_a_merge_empty: sleep 5 2024-09-16T17:18:47.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:70: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change false 2024-09-16T17:18:47.470 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-16T17:18:47.470 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "" 2024-09-16T17:18:47.470 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-16T17:18:47.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:74: TEST_a_merge_empty: ceph osd in 0 1 2024-09-16T17:18:48.052 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already in. osd.1 is already in. 2024-09-16T17:18:48.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:75: TEST_a_merge_empty: ceph osd pool set foo size 3 2024-09-16T17:18:49.443 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 3 2024-09-16T17:18:49.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:77: TEST_a_merge_empty: wait_for_clean 2024-09-16T17:18:49.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:18:49.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:18:49.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:18:49.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:18:49.465 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:18:49.465 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:18:49.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:18:49.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:18:49.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:18:49.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:18:49.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:18:49.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:18:49.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:18:49.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:18:49.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:18:50.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:18:50.138 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:18:50.138 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:18:50.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:18:50.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:50.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:18:50.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836519 2024-09-16T17:18:50.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836519 2024-09-16T17:18:50.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836519' 2024-09-16T17:18:50.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:50.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:18:50.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672992 2024-09-16T17:18:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672992 2024-09-16T17:18:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836519 1-42949672992' 2024-09-16T17:18:50.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:18:50.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:18:50.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018888 2024-09-16T17:18:50.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018888 2024-09-16T17:18:50.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836519 1-42949672992 2-128849018888' 2024-09-16T17:18:50.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:50.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836519 2024-09-16T17:18:50.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:50.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:18:50.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836519 2024-09-16T17:18:50.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:50.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836519 2024-09-16T17:18:50.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-09-16T17:18:50.641 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836519 2024-09-16T17:18:50.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:18:51.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836517 -lt 21474836519 2024-09-16T17:18:51.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:18:52.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:18:52.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:18:52.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836520 -lt 21474836519 2024-09-16T17:18:52.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:52.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672992 2024-09-16T17:18:52.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:52.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:18:52.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:52.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672992 2024-09-16T17:18:52.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672992 2024-09-16T17:18:52.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-16T17:18:52.831 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672992 2024-09-16T17:18:52.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:18:53.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672992 -lt 42949672992 2024-09-16T17:18:53.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:18:53.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-128849018888 2024-09-16T17:18:53.262 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:18:53.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:18:53.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-128849018888 2024-09-16T17:18:53.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:18:53.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018888 2024-09-16T17:18:53.266 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 128849018888 2024-09-16T17:18:53.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 128849018888' 2024-09-16T17:18:53.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:18:53.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018888 -lt 128849018888 2024-09-16T17:18:53.714 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:18:53.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:18:53.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:18:54.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T17:18:54.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:18:54.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:18:54.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:18:54.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:18:54.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:18:54.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:18:54.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:18:54.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T17:18:54.709 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:18:54.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:18:54.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:18:55.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-16T17:18:55.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T17:18:55.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T17:18:55.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T17:18:55.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T17:18:55.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T17:18:55.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T17:18:55.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:18:55.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:18:55.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:18:55.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:18:55.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:18:55.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:18:55.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:18:55.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T17:18:55.770 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:18:55.770 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:18:55.770 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:18:56.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T17:18:56.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:18:56.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:18:56.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:81: TEST_a_merge_empty: ceph pg scrub 1.0 2024-09-16T17:18:56.719 INFO:tasks.workunit.client.0.smithi039.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-16T17:18:56.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:82: TEST_a_merge_empty: sleep 10 2024-09-16T17:19:06.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:83: TEST_a_merge_empty: ceph log last debug 2024-09-16T17:19:07.278 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:48.520432+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e38: 3 total, 3 up, 3 in 2024-09-16T17:19:07.278 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:48.931202+0000 mon.a (mon.0) 406 : cluster [DBG] osdmap e39: 3 total, 3 up, 3 in 2024-09-16T17:19:07.278 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:49.265920+0000 mgr.x (mgr.4100) 75 : cluster [DBG] pgmap v105: 2 pgs: 2 unknown; 90 KiB data, 54 MiB used, 200 GiB / 200 GiB avail 2024-09-16T17:19:07.278 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:49.442826+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e40: 3 total, 3 up, 3 in 2024-09-16T17:19:07.278 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:49.861725+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e41: 3 total, 3 up, 3 in 2024-09-16T17:19:07.278 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:50.205063+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e42: 3 total, 3 up, 3 in 2024-09-16T17:19:07.278 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:50.849743+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e43: 3 total, 3 up, 3 in 2024-09-16T17:19:07.278 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:51.266510+0000 mgr.x (mgr.4100) 76 : cluster [DBG] pgmap v110: 1 pgs: 1 remapped+peering; 141 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 22 KiB/s wr, 1 op/s 2024-09-16T17:19:07.278 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:50.897906+0000 osd.1 (osd.1) 3 : cluster [INF] 1.0 continuing backfill to osd.2 from (24'7,24'7] MIN to 24'7 2024-09-16T17:19:07.279 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:51.746938+0000 mon.a (mon.0) 417 : cluster [DBG] osdmap e44: 3 total, 3 up, 3 in 2024-09-16T17:19:07.279 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:52.190767+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e45: 3 total, 3 up, 3 in 2024-09-16T17:19:07.279 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:53.266848+0000 mgr.x (mgr.4100) 77 : cluster [DBG] pgmap v113: 1 pgs: 1 remapped+peering; 141 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 15 KiB/s wr, 1 op/s 2024-09-16T17:19:07.279 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:55.267369+0000 mgr.x (mgr.4100) 80 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 141 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 10 KiB/s wr, 0 op/s; 10 KiB/s, 0 objects/s recovering 2024-09-16T17:19:07.279 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:57.267891+0000 mgr.x (mgr.4100) 85 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 141 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 8.0 KiB/s, 0 objects/s recovering 2024-09-16T17:19:07.279 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:59.268406+0000 mgr.x (mgr.4100) 86 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 141 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 6.4 KiB/s, 0 objects/s recovering 2024-09-16T17:19:07.279 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:59.642953+0000 osd.1 (osd.1) 4 : cluster [DBG] 1.0 scrub starts 2024-09-16T17:19:07.279 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:18:59.648882+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub ok 2024-09-16T17:19:07.279 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:19:01.268952+0000 mgr.x (mgr.4100) 87 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 141 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 5.4 KiB/s, 0 objects/s recovering 2024-09-16T17:19:07.279 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:19:03.269480+0000 mgr.x (mgr.4100) 88 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 141 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 4.6 KiB/s, 0 objects/s recovering 2024-09-16T17:19:07.279 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-16T17:19:05.270018+0000 mgr.x (mgr.4100) 89 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 141 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 4.3 KiB/s, 0 objects/s recovering 2024-09-16T17:19:07.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:84: TEST_a_merge_empty: ceph pg ls 2024-09-16T17:19:07.720 INFO:tasks.workunit.client.0.smithi039.stdout:PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING 2024-09-16T17:19:07.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 11 0 0 0 144111 0 0 0 11 active+clean 8s 24'7 45:96 [1,0,2]p1 [1,0,2]p1 2024-09-16T17:18:59.648892+0000 2024-09-16T17:18:06.905703+0000 1 periodic scrub scheduled @ 2024-09-18T04:40:14.347341+0000 2024-09-16T17:19:07.721 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-16T17:19:07.721 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T17:19:07.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: grep ' active.clean ' 2024-09-16T17:19:07.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: ceph pg ls 2024-09-16T17:19:08.133 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 11 0 0 0 144111 0 0 0 11 active+clean 8s 24'7 45:96 [1,0,2]p1 [1,0,2]p1 2024-09-16T17:18:59.648892+0000 2024-09-16T17:18:06.905703+0000 1 periodic scrub scheduled @ 2024-09-18T04:40:14.347341+0000 2024-09-16T17:19:08.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-09-16T17:19:08.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-16T17:19:08.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:19:08.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-16T17:19:08.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:19:08.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:19:08.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:19:08.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:19:08.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:19:08.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:19:08.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:19:08.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:19:08.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:19:08.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:19:08.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:19:08.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:19:08.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:19:08.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:19:08.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:19:08.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:19:08.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:19:08.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:19:08.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-16T17:19:08.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:19:08.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:19:08.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:19:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.444977 2024-09-16T17:19:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:19:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:19:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-09-16T17:19:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-09-16T17:19:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-09-16T17:19:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-09-16T17:19:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-16T17:19:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:19:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-16T17:19:08.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:19:08.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:19:08.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:19:08.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:19:08.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:19:08.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:19:08.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:19:08.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:19:08.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:19:08.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:19:08.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:19:08.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:19:08.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:19:08.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:19:08.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:19:08.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:19:08.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:19:08.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:19:08.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-16T17:19:08.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:19:08.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:19:08.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:19:08.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.444977 2024-09-16T17:19:08.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:19:08.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:19:08.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-09-16T17:19:08.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:19:08.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:19:08.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:19:08.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.444977 2024-09-16T17:19:08.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_merge_and_gap td/pg-split-merge 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:89: TEST_import_after_merge_and_gap: local dir=td/pg-split-merge 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:91: TEST_import_after_merge_and_gap: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-09-16T17:19:08.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T17:19:08.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:19:08.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:19:08.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:19:08.918 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:19:08.918 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:19:08.918 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:19:08.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:19:08.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T17:19:09.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:19:09.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:19:09.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:19:09.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:19:09.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:19:09.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:19:09.123 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:19:09.123 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:19:09.123 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:19:09.123 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:19:09.123 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:19:09.123 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:19:09.123 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.444977/ceph-mon.a.asok 2024-09-16T17:19:09.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:19:09.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.444977/ceph-mon.a.asok config get fsid 2024-09-16T17:19:09.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:19:09.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:19:09.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:19:09.242 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:19:09.243 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:19:09.243 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:19:09.243 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:19:09.243 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:19:09.243 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:19:09.243 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:19:09.243 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:19:09.244 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.444977/ceph-mon.a.asok 2024-09-16T17:19:09.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:19:09.244 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.444977/ceph-mon.a.asok config get mon_host 2024-09-16T17:19:09.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:92: TEST_import_after_merge_and_gap: run_mgr td/pg-split-merge x 2024-09-16T17:19:09.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-09-16T17:19:09.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:19:09.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:19:09.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:19:09.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-09-16T17:19:09.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:19:10.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:19:10.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:19:10.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:19:10.260 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:19:10.260 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:19:10.260 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:19:10.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:19:10.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:19:10.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:19:10.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:93: TEST_import_after_merge_and_gap: run_osd td/pg-split-merge 0 2024-09-16T17:19:10.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-16T17:19:10.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:19:10.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:19:10.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:19:10.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-09-16T17:19:10.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:19:10.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:19:10.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:19:10.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-09-16T17:19:10.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:19:10.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=68294d40-1585-4437-81ff-1190991a9a32 2024-09-16T17:19:10.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 68294d40-1585-4437-81ff-1190991a9a32' 2024-09-16T17:19:10.305 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 68294d40-1585-4437-81ff-1190991a9a32 2024-09-16T17:19:10.305 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:19:10.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOaOhmrQcmExAA+Eo0zm/uaQCTN0UK1qG/lw== 2024-09-16T17:19:10.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOaOhmrQcmExAA+Eo0zm/uaQCTN0UK1qG/lw=="}' 2024-09-16T17:19:10.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 68294d40-1585-4437-81ff-1190991a9a32 -i td/pg-split-merge/0/new.json 2024-09-16T17:19:10.684 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:19:10.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-09-16T17:19:10.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOaOhmrQcmExAA+Eo0zm/uaQCTN0UK1qG/lw== --osd-uuid 68294d40-1585-4437-81ff-1190991a9a32 2024-09-16T17:19:10.738 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:10.732+0000 7f1de5e5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:19:10.740 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:10.736+0000 7f1de5e5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:19:10.741 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:10.736+0000 7f1de5e5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:19:10.742 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:10.736+0000 7f1de5e5e7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-16T17:19:10.742 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:10.736+0000 7f1de5e5e7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-16T17:19:10.742 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:10.736+0000 7f1de5e5e7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-16T17:19:10.858 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:10.852+0000 7f1de5e5e7c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-09-16T17:19:17.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-09-16T17:19:17.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:19:17.138 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:19:17.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:19:17.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:19:17.692 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:19:17.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:19:17.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:19:17.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:19:17.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:19:17.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:19:17.726 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:17.724+0000 7f2d96b6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:19:17.749 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:17.744+0000 7f2d96b6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:19:17.750 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:17.744+0000 7f2d96b6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:19:18.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:19:18.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:19:18.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:19:18.113 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:19:18.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:19:18.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:19:18.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:18.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:19:18.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:18.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:18.325 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:18.320+0000 7f2d96b6b7c0 -1 Falling back to public interface 2024-09-16T17:19:18.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:19.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:19.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:19.548 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:19:19.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:19:19.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:19.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:19.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:20.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:20.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:20.976 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:19:20.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:19:20.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:20.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:20.978 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:20.976+0000 7f2d96b6b7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:19:21.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:22.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:22.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:22.576 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:19:22.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:19:22.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:22.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:23.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:24.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:24.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:24.056 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:19:24.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:19:24.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:24.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:24.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:25.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:25.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:25.512 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:19:25.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:19:25.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:25.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:26.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:27.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:27.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:27.026 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:19:27.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:19:27.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:27.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:27.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:28.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:28.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:28.486 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:19:28.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:19:28.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:28.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:29.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:30.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:30.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:30.036 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:19:30.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:19:30.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:30.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:30.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:31.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:31.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:31.527 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:19:31.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:19:31.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:32.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:33.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:33.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:33.003 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:19:33.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:19:33.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:33.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:33.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:34.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:34.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:34.440 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:19:34.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:19:34.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:34.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:34.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:35.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:35.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:35.950 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:19:35.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:19:35.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:35.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:36.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:37.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:37.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:37.697 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:19:37.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:19:37.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:37.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:38.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:39.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:39.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:39.844 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:19:39.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:19:39.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:39.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:40.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:41.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:41.400 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:19:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:19:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:41.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:42.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:43.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:43.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:43.420 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:19:43.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:19:43.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:43.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:44.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:19:45.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:19:45.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:45.605 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:19:45.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:19:45.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:19:45.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:46.040 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1924856843,v1:127.0.0.1:6803/1924856843] [v2:127.0.0.1:6804/1924856843,v1:127.0.0.1:6805/1924856843] exists,up 68294d40-1585-4437-81ff-1190991a9a32 2024-09-16T17:19:46.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:19:46.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:19:46.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:19:46.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:95: TEST_import_after_merge_and_gap: ceph osd pool create foo 2 2024-09-16T17:19:46.677 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T17:19:46.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:96: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-16T17:19:46.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:19:46.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:19:46.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:19:46.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:19:46.696 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:19:46.696 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:19:46.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:19:46.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:19:46.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:19:46.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:19:46.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:19:46.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:19:46.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:19:46.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:19:46.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:19:47.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T17:19:47.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:19:47.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:19:47.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:19:47.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836491 2024-09-16T17:19:47.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836491 2024-09-16T17:19:47.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836491' 2024-09-16T17:19:47.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:19:47.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836491 2024-09-16T17:19:47.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:19:47.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:19:47.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836491 2024-09-16T17:19:47.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:19:47.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836491 2024-09-16T17:19:47.590 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836491 2024-09-16T17:19:47.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-16T17:19:47.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:19:48.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-16T17:19:48.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:19:49.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:19:49.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:19:49.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836491 -lt 21474836491 2024-09-16T17:19:49.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:19:49.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:19:49.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:19:50.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-16T17:19:50.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:19:50.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:19:50.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:19:50.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:19:50.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:19:50.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:19:50.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:19:50.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T17:19:50.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:19:50.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:19:50.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:19:51.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-16T17:19:51.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:19:51.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:19:51.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:97: TEST_import_after_merge_and_gap: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-09-16T17:19:51.039 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-16T17:19:51.039 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-09-16T17:19:51.039 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_455948 2024-09-16T17:19:51.039 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T17:19:51.039 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-16T17:19:52.039 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 96 80 0.0781196 0.078125 0.150796 0.17805 2024-09-16T17:19:53.039 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 208 192 0.0937395 0.109375 0.117548 0.161439 2024-09-16T17:19:54.040 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 320 304 0.0989465 0.109375 0.125475 0.156633 2024-09-16T17:19:55.040 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 3.09282 2024-09-16T17:19:55.040 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 320 2024-09-16T17:19:55.040 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 1024 2024-09-16T17:19:55.040 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 1024 2024-09-16T17:19:55.040 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.101041 2024-09-16T17:19:55.040 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.0180422 2024-09-16T17:19:55.040 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.109375 2024-09-16T17:19:55.040 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.078125 2024-09-16T17:19:55.040 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 103 2024-09-16T17:19:55.041 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 18.4752 2024-09-16T17:19:55.041 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 112 2024-09-16T17:19:55.041 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 80 2024-09-16T17:19:55.041 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.154288 2024-09-16T17:19:55.041 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.0472006 2024-09-16T17:19:55.041 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 0.309407 2024-09-16T17:19:55.041 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.108423 2024-09-16T17:19:55.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:99: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-09-16T17:19:55.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:19:55.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:19:55.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:19:55.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:19:55.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:19:56.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:19:56.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:100: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-09-16T17:19:57.865 INFO:tasks.workunit.client.0.smithi039.stderr:Exporting 1.1 info 1.1( v 9'160 (0'0,9'160] local-lis/les=7/8 n=160 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-16T17:19:57.869 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:81444e03:::benchmark_data_smithi039_455948_object191:head# 2024-09-16T17:19:57.869 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:81524550:::benchmark_data_smithi039_455948_object291:head# 2024-09-16T17:19:57.869 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:81688782:::benchmark_data_smithi039_455948_object114:head# 2024-09-16T17:19:57.880 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8340282c:::benchmark_data_smithi039_455948_object261:head# 2024-09-16T17:19:57.880 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:84461d11:::benchmark_data_smithi039_455948_object236:head# 2024-09-16T17:19:57.882 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:85bbfc69:::benchmark_data_smithi039_455948_object30:head# 2024-09-16T17:19:57.883 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:861aea5f:::benchmark_data_smithi039_455948_object258:head# 2024-09-16T17:19:57.883 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:869e8a5e:::benchmark_data_smithi039_455948_object167:head# 2024-09-16T17:19:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:872ef1cc:::benchmark_data_smithi039_455948_object195:head# 2024-09-16T17:19:57.884 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8923ed3b:::benchmark_data_smithi039_455948_object5:head# 2024-09-16T17:19:57.885 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8a3ec687:::benchmark_data_smithi039_455948_object214:head# 2024-09-16T17:19:57.885 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8aec744d:::benchmark_data_smithi039_455948_object254:head# 2024-09-16T17:19:57.886 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8c74c0d6:::benchmark_data_smithi039_455948_object116:head# 2024-09-16T17:19:57.887 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8eb4657a:::benchmark_data_smithi039_455948_object163:head# 2024-09-16T17:19:57.887 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92b8125e:::benchmark_data_smithi039_455948_object165:head# 2024-09-16T17:19:57.888 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92dd79ef:::benchmark_data_smithi039_455948_object130:head# 2024-09-16T17:19:57.888 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92ec70a0:::benchmark_data_smithi039_455948_object170:head# 2024-09-16T17:19:57.889 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92f0e148:::benchmark_data_smithi039_455948_object223:head# 2024-09-16T17:19:57.889 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:930eb7ac:::benchmark_data_smithi039_455948_object262:head# 2024-09-16T17:19:57.890 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:972245e1:::benchmark_data_smithi039_455948_object7:head# 2024-09-16T17:19:57.891 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:972cbf49:::benchmark_data_smithi039_455948_object3:head# 2024-09-16T17:19:57.891 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:984c6070:::benchmark_data_smithi039_455948_object1:head# 2024-09-16T17:19:57.892 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:98946638:::benchmark_data_smithi039_455948_object219:head# 2024-09-16T17:19:57.892 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:98a59f74:::benchmark_data_smithi039_455948_object216:head# 2024-09-16T17:19:57.893 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:98b0eb5d:::benchmark_data_smithi039_455948_object131:head# 2024-09-16T17:19:57.893 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:99654740:::benchmark_data_smithi039_455948_object141:head# 2024-09-16T17:19:57.894 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9b110e6f:::benchmark_data_smithi039_455948_object55:head# 2024-09-16T17:19:57.894 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9c86eb42:::benchmark_data_smithi039_455948_object287:head# 2024-09-16T17:19:57.895 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9c901805:::benchmark_data_smithi039_455948_object59:head# 2024-09-16T17:19:57.895 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9d050ed9:::benchmark_data_smithi039_455948_object284:head# 2024-09-16T17:19:57.896 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9e116042:::benchmark_data_smithi039_455948_object286:head# 2024-09-16T17:19:57.897 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9eac4879:::benchmark_data_smithi039_455948_object289:head# 2024-09-16T17:19:57.897 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9f0bc29c:::benchmark_data_smithi039_455948_object257:head# 2024-09-16T17:19:57.898 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a0acf962:::benchmark_data_smithi039_455948_object61:head# 2024-09-16T17:19:57.898 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a0b1ff33:::benchmark_data_smithi039_455948_object133:head# 2024-09-16T17:19:57.899 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a299f295:::benchmark_data_smithi039_455948_object176:head# 2024-09-16T17:19:57.899 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a3610da1:::benchmark_data_smithi039_455948_object149:head# 2024-09-16T17:19:57.899 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a3616c00:::benchmark_data_smithi039_455948_object246:head# 2024-09-16T17:19:57.900 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a3d8a56b:::benchmark_data_smithi039_455948_object73:head# 2024-09-16T17:19:57.901 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a437720d:::benchmark_data_smithi039_455948_object13:head# 2024-09-16T17:19:57.902 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a5276dd6:::benchmark_data_smithi039_455948_object2:head# 2024-09-16T17:19:57.902 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a5e30e12:::benchmark_data_smithi039_455948_object17:head# 2024-09-16T17:19:57.902 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a5e964e3:::benchmark_data_smithi039_455948_object45:head# 2024-09-16T17:19:57.902 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a5f6a828:::benchmark_data_smithi039_455948_object25:head# 2024-09-16T17:19:57.903 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a63079ff:::benchmark_data_smithi039_455948_object265:head# 2024-09-16T17:19:57.903 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a64c1b86:::benchmark_data_smithi039_455948_object288:head# 2024-09-16T17:19:57.904 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a6698c47:::benchmark_data_smithi039_455948_object259:head# 2024-09-16T17:19:57.905 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a69d0e24:::benchmark_data_smithi039_455948_object113:head# 2024-09-16T17:19:57.913 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a6c2f07b:::benchmark_data_smithi039_455948_object212:head# 2024-09-16T17:19:57.915 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a8211e63:::benchmark_data_smithi039_455948_object281:head# 2024-09-16T17:19:57.917 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ad13fc26:::benchmark_data_smithi039_455948_object110:head# 2024-09-16T17:19:57.917 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ad8239d1:::benchmark_data_smithi039_455948_object232:head# 2024-09-16T17:19:57.918 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ad91945e:::benchmark_data_smithi039_455948_object93:head# 2024-09-16T17:19:57.918 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:adaffd31:::benchmark_data_smithi039_455948_object26:head# 2024-09-16T17:19:57.919 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:aeb14177:::benchmark_data_smithi039_455948_object29:head# 2024-09-16T17:19:57.919 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:aecd8d89:::benchmark_data_smithi039_455948_object174:head# 2024-09-16T17:19:57.919 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b03ef840:::benchmark_data_smithi039_455948_object295:head# 2024-09-16T17:19:57.920 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b0c8cac8:::benchmark_data_smithi039_455948_object117:head# 2024-09-16T17:19:57.920 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b137eb01:::benchmark_data_smithi039_455948_object211:head# 2024-09-16T17:19:57.921 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b2347b11:::benchmark_data_smithi039_455948_object301:head# 2024-09-16T17:19:57.921 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b249856d:::benchmark_data_smithi039_455948_object235:head# 2024-09-16T17:19:57.921 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b3253c9c:::benchmark_data_smithi039_455948_object255:head# 2024-09-16T17:19:57.922 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b5b57515:::benchmark_data_smithi039_455948_object129:head# 2024-09-16T17:19:57.922 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b6c46095:::benchmark_data_smithi039_455948_object178:head# 2024-09-16T17:19:57.922 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b6ce9049:::benchmark_data_smithi039_455948_object172:head# 2024-09-16T17:19:57.923 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b6cf5c8c:::benchmark_data_smithi039_455948_object231:head# 2024-09-16T17:19:57.923 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b7c8f422:::benchmark_data_smithi039_455948_object112:head# 2024-09-16T17:19:57.924 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-09-16T17:19:57.924 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b8b98acb:::benchmark_data_smithi039_455948_object313:head# 2024-09-16T17:19:57.925 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b9991d10:::benchmark_data_smithi039_455948_object154:head# 2024-09-16T17:19:57.925 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:baf26fd7:::benchmark_data_smithi039_455948_object134:head# 2024-09-16T17:19:57.925 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bb71b590:::benchmark_data_smithi039_455948_object314:head# 2024-09-16T17:19:57.925 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bc052407:::benchmark_data_smithi039_455948_object270:head# 2024-09-16T17:19:57.926 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bcff6325:::benchmark_data_smithi039_455948_object315:head# 2024-09-16T17:19:57.926 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bd19ab08:::benchmark_data_smithi039_455948_object81:head# 2024-09-16T17:19:57.926 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:be7390f5:::benchmark_data_smithi039_455948_object83:head# 2024-09-16T17:19:57.927 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c07324f3:::benchmark_data_smithi039_455948_object302:head# 2024-09-16T17:19:57.927 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c0d972fb:::benchmark_data_smithi039_455948_object209:head# 2024-09-16T17:19:57.928 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c34a47c1:::benchmark_data_smithi039_455948_object319:head# 2024-09-16T17:19:57.928 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c38689dc:::benchmark_data_smithi039_455948_object50:head# 2024-09-16T17:19:57.929 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c3893c52:::benchmark_data_smithi039_455948_object66:head# 2024-09-16T17:19:57.929 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c464c3ce:::benchmark_data_smithi039_455948_object106:head# 2024-09-16T17:19:57.930 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c56ee08e:::benchmark_data_smithi039_455948_object125:head# 2024-09-16T17:19:57.931 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c6555459:::benchmark_data_smithi039_455948_object282:head# 2024-09-16T17:19:57.931 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c70cd8e6:::benchmark_data_smithi039_455948_object181:head# 2024-09-16T17:19:57.932 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c72b9798:::benchmark_data_smithi039_455948_object186:head# 2024-09-16T17:19:57.932 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c7547bd4:::benchmark_data_smithi039_455948_object160:head# 2024-09-16T17:19:57.933 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c8d50a66:::benchmark_data_smithi039_455948_object15:head# 2024-09-16T17:19:57.933 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c916bf36:::benchmark_data_smithi039_455948_object300:head# 2024-09-16T17:19:57.934 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c98347ce:::benchmark_data_smithi039_455948_object44:head# 2024-09-16T17:19:57.935 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ca283ebf:::benchmark_data_smithi039_455948_object245:head# 2024-09-16T17:19:57.935 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cb190b2d:::benchmark_data_smithi039_455948_object299:head# 2024-09-16T17:19:57.936 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cb44df0b:::benchmark_data_smithi039_455948_object102:head# 2024-09-16T17:19:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cb8dc5ee:::benchmark_data_smithi039_455948_object201:head# 2024-09-16T17:19:57.937 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cd24ea32:::benchmark_data_smithi039_455948_object42:head# 2024-09-16T17:19:57.938 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cd94f270:::benchmark_data_smithi039_455948_object124:head# 2024-09-16T17:19:57.939 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ce974f23:::benchmark_data_smithi039_455948_object159:head# 2024-09-16T17:19:57.939 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cf1dabff:::benchmark_data_smithi039_455948_object151:head# 2024-09-16T17:19:57.939 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cf56f22e:::benchmark_data_smithi039_455948_object294:head# 2024-09-16T17:19:57.940 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d04eceaf:::benchmark_data_smithi039_455948_object39:head# 2024-09-16T17:19:57.941 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d06d3616:::benchmark_data_smithi039_455948_object24:head# 2024-09-16T17:19:57.942 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d08457ea:::benchmark_data_smithi039_455948_object4:head# 2024-09-16T17:19:57.942 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d0851a13:::benchmark_data_smithi039_455948_object306:head# 2024-09-16T17:19:57.942 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d08dfc7d:::benchmark_data_smithi039_455948_object89:head# 2024-09-16T17:19:57.943 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d1c0a25d:::benchmark_data_smithi039_455948_object105:head# 2024-09-16T17:19:57.944 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d2347595:::benchmark_data_smithi039_455948_object123:head# 2024-09-16T17:19:57.944 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d2f20841:::benchmark_data_smithi039_455948_object95:head# 2024-09-16T17:19:57.945 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d33b721d:::benchmark_data_smithi039_455948_object244:head# 2024-09-16T17:19:57.946 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d37b01a7:::benchmark_data_smithi039_455948_object243:head# 2024-09-16T17:19:57.946 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d432d0ae:::benchmark_data_smithi039_455948_object99:head# 2024-09-16T17:19:57.947 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d4cfa604:::benchmark_data_smithi039_455948_object14:head# 2024-09-16T17:19:57.948 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d66a0831:::benchmark_data_smithi039_455948_object143:head# 2024-09-16T17:19:57.948 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d7dff5fe:::benchmark_data_smithi039_455948_object51:head# 2024-09-16T17:19:57.949 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d81d2707:::benchmark_data_smithi039_455948_object248:head# 2024-09-16T17:19:57.950 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d8cdeb85:::benchmark_data_smithi039_455948_object279:head# 2024-09-16T17:19:57.950 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d95fa43b:::benchmark_data_smithi039_455948_object97:head# 2024-09-16T17:19:57.951 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:db0a03c3:::benchmark_data_smithi039_455948_object35:head# 2024-09-16T17:19:57.951 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dbe2ac53:::benchmark_data_smithi039_455948_object92:head# 2024-09-16T17:19:57.952 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e05e2d0e:::benchmark_data_smithi039_455948_object127:head# 2024-09-16T17:19:57.953 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e18f9e35:::benchmark_data_smithi039_455948_object82:head# 2024-09-16T17:19:57.953 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e1a4e1b7:::benchmark_data_smithi039_455948_object189:head# 2024-09-16T17:19:57.953 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e2081651:::benchmark_data_smithi039_455948_object303:head# 2024-09-16T17:19:57.954 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e36479d3:::benchmark_data_smithi039_455948_object277:head# 2024-09-16T17:19:57.954 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e4509956:::benchmark_data_smithi039_455948_object222:head# 2024-09-16T17:19:57.955 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e45ad9a8:::benchmark_data_smithi039_455948_object310:head# 2024-09-16T17:19:57.955 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e49f1f25:::benchmark_data_smithi039_455948_object60:head# 2024-09-16T17:19:57.956 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e8279365:::benchmark_data_smithi039_455948_object162:head# 2024-09-16T17:19:57.957 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e84b1af3:::benchmark_data_smithi039_455948_object168:head# 2024-09-16T17:19:57.957 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e860450e:::benchmark_data_smithi039_455948_object40:head# 2024-09-16T17:19:57.958 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e9546805:::benchmark_data_smithi039_455948_object84:head# 2024-09-16T17:19:57.958 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e9c31f48:::benchmark_data_smithi039_455948_object57:head# 2024-09-16T17:19:57.959 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e9db3ae8:::benchmark_data_smithi039_455948_object185:head# 2024-09-16T17:19:57.960 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ea00b668:::benchmark_data_smithi039_455948_object126:head# 2024-09-16T17:19:57.961 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:eac552d0:::benchmark_data_smithi039_455948_object217:head# 2024-09-16T17:19:57.961 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ed7c17e0:::benchmark_data_smithi039_455948_object152:head# 2024-09-16T17:19:57.962 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:edc14e53:::benchmark_data_smithi039_455948_object230:head# 2024-09-16T17:19:57.962 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:efd04b1c:::benchmark_data_smithi039_455948_object225:head# 2024-09-16T17:19:57.963 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:efea2d35:::benchmark_data_smithi039_455948_object146:head# 2024-09-16T17:19:57.964 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f1bb07dd:::benchmark_data_smithi039_455948_object298:head# 2024-09-16T17:19:57.964 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f1e99f50:::benchmark_data_smithi039_455948_object215:head# 2024-09-16T17:19:57.964 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f1f5e565:::benchmark_data_smithi039_455948_object307:head# 2024-09-16T17:19:57.965 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f20c71ff:::benchmark_data_smithi039_455948_object267:head# 2024-09-16T17:19:57.965 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f274efb2:::benchmark_data_smithi039_455948_object27:head# 2024-09-16T17:19:57.967 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f2a8576b:::benchmark_data_smithi039_455948_object54:head# 2024-09-16T17:19:57.967 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f2f0a09c:::benchmark_data_smithi039_455948_object227:head# 2024-09-16T17:19:57.967 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f5996306:::benchmark_data_smithi039_455948_object22:head# 2024-09-16T17:19:57.967 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f61ef65d:::benchmark_data_smithi039_455948_object111:head# 2024-09-16T17:19:57.968 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f621bc7f:::benchmark_data_smithi039_455948_object182:head# 2024-09-16T17:19:57.969 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f72c3135:::benchmark_data_smithi039_455948_object21:head# 2024-09-16T17:19:57.969 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f7b30eee:::benchmark_data_smithi039_455948_object194:head# 2024-09-16T17:19:57.970 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f97e448b:::benchmark_data_smithi039_455948_object10:head# 2024-09-16T17:19:57.970 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f9d8cd97:::benchmark_data_smithi039_455948_object199:head# 2024-09-16T17:19:57.971 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f9e3593c:::benchmark_data_smithi039_455948_object256:head# 2024-09-16T17:19:57.972 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fa9bf67e:::benchmark_data_smithi039_455948_object32:head# 2024-09-16T17:19:57.972 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fadac265:::benchmark_data_smithi039_455948_object305:head# 2024-09-16T17:19:57.973 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fadc5b0b:::benchmark_data_smithi039_455948_object104:head# 2024-09-16T17:19:57.973 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fd05df8b:::benchmark_data_smithi039_455948_object70:head# 2024-09-16T17:19:57.974 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fdea465d:::benchmark_data_smithi039_455948_object107:head# 2024-09-16T17:19:57.974 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fe001570:::benchmark_data_smithi039_455948_object171:head# 2024-09-16T17:19:57.975 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ff9caaec:::benchmark_data_smithi039_455948_object263:head# 2024-09-16T17:19:57.976 INFO:tasks.workunit.client.0.smithi039.stderr:Export successful 2024-09-16T17:19:58.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:101: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-09-16T17:19:59.051 INFO:tasks.workunit.client.0.smithi039.stderr:Exporting 1.0 info 1.0( v 9'161 (0'0,9'161] local-lis/les=7/8 n=161 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-16T17:19:59.054 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:00456576:::benchmark_data_smithi039_455948_object157:head# 2024-09-16T17:19:59.054 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:014ebcb1:::benchmark_data_smithi039_455948_object233:head# 2024-09-16T17:19:59.055 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:01dc36eb:::benchmark_data_smithi039_455948_object64:head# 2024-09-16T17:19:59.056 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:01f0fd22:::benchmark_data_smithi039_455948_object208:head# 2024-09-16T17:19:59.056 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:01f5a6f9:::benchmark_data_smithi039_455948_object272:head# 2024-09-16T17:19:59.057 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:02207eee:::benchmark_data_smithi039_455948_object192:head# 2024-09-16T17:19:59.057 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:02bddb95:::benchmark_data_smithi039_455948_object137:head# 2024-09-16T17:19:59.058 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:033989ba:::benchmark_data_smithi039_455948_object80:head# 2024-09-16T17:19:59.058 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:041f4598:::benchmark_data_smithi039_455948_object187:head# 2024-09-16T17:19:59.059 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:047a0c2d:::benchmark_data_smithi039_455948_object293:head# 2024-09-16T17:19:59.060 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:04aae244:::benchmark_data_smithi039_455948_object98:head# 2024-09-16T17:19:59.060 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:058f836c:::benchmark_data_smithi039_455948_object205:head# 2024-09-16T17:19:59.060 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:05b9dc7f:::benchmark_data_smithi039_455948_object184:head# 2024-09-16T17:19:59.061 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:05f32019:::benchmark_data_smithi039_455948_object180:head# 2024-09-16T17:19:59.062 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0a2f45e2:::benchmark_data_smithi039_455948_object136:head# 2024-09-16T17:19:59.062 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0a3ddd05:::benchmark_data_smithi039_455948_object229:head# 2024-09-16T17:19:59.063 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0dc71e56:::benchmark_data_smithi039_455948_object224:head# 2024-09-16T17:19:59.063 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0e79e577:::benchmark_data_smithi039_455948_object308:head# 2024-09-16T17:19:59.064 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0e80050a:::benchmark_data_smithi039_455948_object9:head# 2024-09-16T17:19:59.064 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0f01eefd:::benchmark_data_smithi039_455948_object166:head# 2024-09-16T17:19:59.065 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0fcc09dc:::benchmark_data_smithi039_455948_object175:head# 2024-09-16T17:19:59.065 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:100e9870:::benchmark_data_smithi039_455948_object122:head# 2024-09-16T17:19:59.066 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:12727e65:::benchmark_data_smithi039_455948_object164:head# 2024-09-16T17:19:59.066 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:12810ad7:::benchmark_data_smithi039_455948_object132:head# 2024-09-16T17:19:59.067 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:12cae338:::benchmark_data_smithi039_455948_object213:head# 2024-09-16T17:19:59.067 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:13b7075e:::benchmark_data_smithi039_455948_object292:head# 2024-09-16T17:19:59.068 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:14072aff:::benchmark_data_smithi039_455948_object266:head# 2024-09-16T17:19:59.069 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:14a5c78e:::benchmark_data_smithi039_455948_object46:head# 2024-09-16T17:19:59.069 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1561ff76:::benchmark_data_smithi039_455948_object155:head# 2024-09-16T17:19:59.070 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:16d84de3:::benchmark_data_smithi039_455948_object200:head# 2024-09-16T17:19:59.070 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1a3cf39c:::benchmark_data_smithi039_455948_object177:head# 2024-09-16T17:19:59.071 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1ae97a14:::benchmark_data_smithi039_455948_object69:head# 2024-09-16T17:19:59.071 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1df9e5c8:::benchmark_data_smithi039_455948_object115:head# 2024-09-16T17:19:59.071 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1e6aee90:::benchmark_data_smithi039_455948_object312:head# 2024-09-16T17:19:59.072 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1e9dda24:::benchmark_data_smithi039_455948_object173:head# 2024-09-16T17:19:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1ea64c7e:::benchmark_data_smithi039_455948_object278:head# 2024-09-16T17:19:59.073 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1fb0f211:::benchmark_data_smithi039_455948_object109:head# 2024-09-16T17:19:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1fbf7b6c:::benchmark_data_smithi039_455948_object207:head# 2024-09-16T17:19:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1fd4abd6:::benchmark_data_smithi039_455948_object142:head# 2024-09-16T17:19:59.075 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:20997167:::benchmark_data_smithi039_455948_object56:head# 2024-09-16T17:19:59.075 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:227f4aa5:::benchmark_data_smithi039_455948_object260:head# 2024-09-16T17:19:59.076 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:241f88ec:::benchmark_data_smithi039_455948_object118:head# 2024-09-16T17:19:59.076 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2464192d:::benchmark_data_smithi039_455948_object67:head# 2024-09-16T17:19:59.077 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:255d9356:::benchmark_data_smithi039_455948_object19:head# 2024-09-16T17:19:59.077 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:260b2bbc:::benchmark_data_smithi039_455948_object79:head# 2024-09-16T17:19:59.078 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:27914c55:::benchmark_data_smithi039_455948_object135:head# 2024-09-16T17:19:59.078 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:279e6bbf:::benchmark_data_smithi039_455948_object247:head# 2024-09-16T17:19:59.079 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:28d56bea:::benchmark_data_smithi039_455948_object11:head# 2024-09-16T17:19:59.079 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:294aa246:::benchmark_data_smithi039_455948_object38:head# 2024-09-16T17:19:59.080 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2a8d03cc:::benchmark_data_smithi039_455948_object197:head# 2024-09-16T17:19:59.080 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2cdc5a61:::benchmark_data_smithi039_455948_object87:head# 2024-09-16T17:19:59.081 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2f2af160:::benchmark_data_smithi039_455948_object63:head# 2024-09-16T17:19:59.081 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2f6ad66d:::benchmark_data_smithi039_455948_object237:head# 2024-09-16T17:19:59.082 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2fa31139:::benchmark_data_smithi039_455948_object36:head# 2024-09-16T17:19:59.082 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2faa5036:::benchmark_data_smithi039_455948_object43:head# 2024-09-16T17:19:59.083 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3061c169:::benchmark_data_smithi039_455948_object271:head# 2024-09-16T17:19:59.083 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:30da3c6c:::benchmark_data_smithi039_455948_object193:head# 2024-09-16T17:19:59.084 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3105728e:::benchmark_data_smithi039_455948_object148:head# 2024-09-16T17:19:59.084 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3173dfe0:::benchmark_data_smithi039_455948_object188:head# 2024-09-16T17:19:59.085 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:335c1a68:::benchmark_data_smithi039_455948_object128:head# 2024-09-16T17:19:59.086 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:351c3af9:::benchmark_data_smithi039_455948_object72:head# 2024-09-16T17:19:59.086 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:36bb23c0:::benchmark_data_smithi039_455948_object101:head# 2024-09-16T17:19:59.086 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:375cb5d3:::benchmark_data_smithi039_455948_object275:head# 2024-09-16T17:19:59.087 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:37a5f014:::benchmark_data_smithi039_455948_object204:head# 2024-09-16T17:19:59.087 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3a6cafe9:::benchmark_data_smithi039_455948_object6:head# 2024-09-16T17:19:59.088 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3c81aa17:::benchmark_data_smithi039_455948_object280:head# 2024-09-16T17:19:59.088 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3cd1f324:::benchmark_data_smithi039_455948_object202:head# 2024-09-16T17:19:59.089 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3eaccc36:::benchmark_data_smithi039_455948_object144:head# 2024-09-16T17:19:59.090 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3f37ec22:::benchmark_data_smithi039_455948_object161:head# 2024-09-16T17:19:59.090 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3fc02a40:::benchmark_data_smithi039_455948_object74:head# 2024-09-16T17:19:59.090 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4003301f:::benchmark_data_smithi039_455948_object139:head# 2024-09-16T17:19:59.091 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4117f431:::benchmark_data_smithi039_455948_object234:head# 2024-09-16T17:19:59.091 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:41facc71:::benchmark_data_smithi039_455948_object198:head# 2024-09-16T17:19:59.092 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4411f440:::benchmark_data_smithi039_455948_object297:head# 2024-09-16T17:19:59.093 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:44802c6d:::benchmark_data_smithi039_455948_object119:head# 2024-09-16T17:19:59.093 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:45c521fb:::benchmark_data_smithi039_455948_object249:head# 2024-09-16T17:19:59.094 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:469ace3a:::benchmark_data_smithi039_455948_object206:head# 2024-09-16T17:19:59.095 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:47970e62:::benchmark_data_smithi039_455948_object210:head# 2024-09-16T17:19:59.095 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:47b13862:::benchmark_data_smithi039_455948_object138:head# 2024-09-16T17:19:59.095 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4801d23d:::benchmark_data_smithi039_455948_object85:head# 2024-09-16T17:19:59.095 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:491895da:::benchmark_data_smithi039_455948_object169:head# 2024-09-16T17:19:59.096 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4935d551:::benchmark_data_smithi039_455948_object251:head# 2024-09-16T17:19:59.097 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:496b18ad:::benchmark_data_smithi039_455948_object253:head# 2024-09-16T17:19:59.097 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4a184ab0:::benchmark_data_smithi039_455948_object158:head# 2024-09-16T17:19:59.098 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4a1d7637:::benchmark_data_smithi039_455948_object183:head# 2024-09-16T17:19:59.098 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4ac873c7:::benchmark_data_smithi039_455948_object37:head# 2024-09-16T17:19:59.099 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4bc4a2e0:::benchmark_data_smithi039_455948_object49:head# 2024-09-16T17:19:59.099 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4c5a7992:::benchmark_data_smithi039_455948_object221:head# 2024-09-16T17:19:59.099 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4f03ab2e:::benchmark_data_smithi039_455948_object318:head# 2024-09-16T17:19:59.100 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4f301666:::benchmark_data_smithi039_455948_object228:head# 2024-09-16T17:19:59.100 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4f3a0a7a:::benchmark_data_smithi039_455948_object240:head# 2024-09-16T17:19:59.101 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4fb8888a:::benchmark_data_smithi039_455948_object203:head# 2024-09-16T17:19:59.101 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:50a96376:::benchmark_data_smithi039_455948_object96:head# 2024-09-16T17:19:59.102 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5244e47b:::benchmark_data_smithi039_455948_object62:head# 2024-09-16T17:19:59.102 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:52d42a8e:::benchmark_data_smithi039_455948_object48:head# 2024-09-16T17:19:59.103 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:52ede5be:::benchmark_data_smithi039_455948_object12:head# 2024-09-16T17:19:59.103 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:53a105b2:::benchmark_data_smithi039_455948_object311:head# 2024-09-16T17:19:59.104 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:53cb9fd2:::benchmark_data_smithi039_455948_object190:head# 2024-09-16T17:19:59.105 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:53ed147c:::benchmark_data_smithi039_455948_object31:head# 2024-09-16T17:19:59.105 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:54d49a7f:::benchmark_data_smithi039_455948_object28:head# 2024-09-16T17:19:59.106 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:54e04d99:::benchmark_data_smithi039_455948_object283:head# 2024-09-16T17:19:59.106 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:54e85f22:::benchmark_data_smithi039_455948_object23:head# 2024-09-16T17:19:59.107 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5651eea5:::benchmark_data_smithi039_455948_object18:head# 2024-09-16T17:19:59.107 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:57dabe70:::benchmark_data_smithi039_455948_object218:head# 2024-09-16T17:19:59.108 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5801fe81:::benchmark_data_smithi039_455948_object75:head# 2024-09-16T17:19:59.109 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:58d2c70a:::benchmark_data_smithi039_455948_object0:head# 2024-09-16T17:19:59.109 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:59614fc8:::benchmark_data_smithi039_455948_object76:head# 2024-09-16T17:19:59.110 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5a3d91e5:::benchmark_data_smithi039_455948_object108:head# 2024-09-16T17:19:59.110 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5a7ae6c2:::benchmark_data_smithi039_455948_object100:head# 2024-09-16T17:19:59.110 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5c28febb:::benchmark_data_smithi039_455948_object156:head# 2024-09-16T17:19:59.111 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5db907e9:::benchmark_data_smithi039_455948_object121:head# 2024-09-16T17:19:59.111 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5ded13e4:::benchmark_data_smithi039_455948_object90:head# 2024-09-16T17:19:59.111 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5e428851:::benchmark_data_smithi039_455948_object309:head# 2024-09-16T17:19:59.112 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5e629290:::benchmark_data_smithi039_455948_object20:head# 2024-09-16T17:19:59.113 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5efeb4ae:::benchmark_data_smithi039_455948_object153:head# 2024-09-16T17:19:59.113 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5ff59d48:::benchmark_data_smithi039_455948_object16:head# 2024-09-16T17:19:59.114 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6137bcba:::benchmark_data_smithi039_455948_object120:head# 2024-09-16T17:19:59.114 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6154b590:::benchmark_data_smithi039_455948_object58:head# 2024-09-16T17:19:59.115 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6280b741:::benchmark_data_smithi039_455948_object250:head# 2024-09-16T17:19:59.115 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:62961663:::benchmark_data_smithi039_455948_object296:head# 2024-09-16T17:19:59.116 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:642ab5c2:::benchmark_data_smithi039_455948_object41:head# 2024-09-16T17:19:59.117 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:64aa8ba8:::benchmark_data_smithi039_455948_object71:head# 2024-09-16T17:19:59.117 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:654ddc28:::benchmark_data_smithi039_455948_object147:head# 2024-09-16T17:19:59.117 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:656a01f0:::benchmark_data_smithi039_455948_object316:head# 2024-09-16T17:19:59.118 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6667fb4d:::benchmark_data_smithi039_455948_object252:head# 2024-09-16T17:19:59.118 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:66719837:::benchmark_data_smithi039_455948_object179:head# 2024-09-16T17:19:59.119 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:67fc60d1:::benchmark_data_smithi039_455948_object103:head# 2024-09-16T17:19:59.119 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:67fc91c1:::benchmark_data_smithi039_455948_object52:head# 2024-09-16T17:19:59.120 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6815ad2e:::benchmark_data_smithi039_455948_object196:head# 2024-09-16T17:19:59.121 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:69293e2b:::benchmark_data_smithi039_455948_object290:head# 2024-09-16T17:19:59.121 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6ae4f661:::benchmark_data_smithi039_455948_object241:head# 2024-09-16T17:19:59.122 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6cb11b42:::benchmark_data_smithi039_455948_object285:head# 2024-09-16T17:19:59.123 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6ce0dbf0:::benchmark_data_smithi039_455948_object268:head# 2024-09-16T17:19:59.123 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6d2fe8c5:::benchmark_data_smithi039_455948_object273:head# 2024-09-16T17:19:59.124 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6d540a4e:::benchmark_data_smithi039_455948_object8:head# 2024-09-16T17:19:59.124 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6d724163:::benchmark_data_smithi039_455948_object269:head# 2024-09-16T17:19:59.125 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6e61b9d5:::benchmark_data_smithi039_455948_object68:head# 2024-09-16T17:19:59.125 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6f095114:::benchmark_data_smithi039_455948_object94:head# 2024-09-16T17:19:59.126 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:70691fc8:::benchmark_data_smithi039_455948_object145:head# 2024-09-16T17:19:59.126 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:708034b3:::benchmark_data_smithi039_455948_object276:head# 2024-09-16T17:19:59.127 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:70f55f31:::benchmark_data_smithi039_455948_object239:head# 2024-09-16T17:19:59.128 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:71b1cb82:::benchmark_data_smithi039_455948_object65:head# 2024-09-16T17:19:59.128 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:71ce8906:::benchmark_data_smithi039_455948_object78:head# 2024-09-16T17:19:59.143 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:72fdddd5:::benchmark_data_smithi039_455948_object86:head# 2024-09-16T17:19:59.157 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:73b633f2:::benchmark_data_smithi039_455948_object150:head# 2024-09-16T17:19:59.157 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:754b4658:::benchmark_data_smithi039_455948_object77:head# 2024-09-16T17:19:59.177 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:75a63f9f:::benchmark_data_smithi039_455948_object226:head# 2024-09-16T17:19:59.190 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:75f9bf67:::benchmark_data_smithi039_455948_object242:head# 2024-09-16T17:19:59.191 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:764ab525:::benchmark_data_smithi039_455948_object317:head# 2024-09-16T17:19:59.191 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:768e2761:::benchmark_data_smithi039_455948_object238:head# 2024-09-16T17:19:59.192 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:77182e0f:::benchmark_data_smithi039_455948_object140:head# 2024-09-16T17:19:59.192 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7739f249:::benchmark_data_smithi039_455948_object34:head# 2024-09-16T17:19:59.193 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:784592ec:::benchmark_data_smithi039_455948_object33:head# 2024-09-16T17:19:59.193 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7a597a33:::benchmark_data_smithi039_455948_object274:head# 2024-09-16T17:19:59.194 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7b291d50:::benchmark_data_smithi039_455948_object220:head# 2024-09-16T17:19:59.194 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7c07aaad:::benchmark_data_smithi039_455948_object91:head# 2024-09-16T17:19:59.195 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7cd2d272:::benchmark_data_smithi039_455948_object53:head# 2024-09-16T17:19:59.195 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7d0d8840:::benchmark_data_smithi039_455948_object264:head# 2024-09-16T17:19:59.196 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7d342ae1:::benchmark_data_smithi039_455948_object47:head# 2024-09-16T17:19:59.196 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7ee172f3:::benchmark_data_smithi039_455948_object304:head# 2024-09-16T17:19:59.197 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7fe7c24d:::benchmark_data_smithi039_455948_object88:head# 2024-09-16T17:19:59.197 INFO:tasks.workunit.client.0.smithi039.stderr:Export successful 2024-09-16T17:19:59.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:102: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-09-16T17:19:59.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-16T17:19:59.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T17:19:59.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T17:19:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T17:19:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-16T17:19:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:19:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:19:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:19:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:19:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-16T17:19:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-16T17:19:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T17:19:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T17:19:59.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-16T17:19:59.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T17:19:59.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:19:59.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:19:59.489 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:19:59.489 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:19:59.489 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:19:59.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:19:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:19:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:19:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-16T17:19:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-16T17:19:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:19:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:19:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:19:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:19:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T17:19:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T17:19:59.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-16T17:19:59.491 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:19:59.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T17:19:59.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:19:59.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-16T17:19:59.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T17:19:59.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T17:19:59.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T17:19:59.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T17:19:59.528 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:59.523+0000 7f1fd64e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:19:59.531 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:59.527+0000 7f1fd64e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:19:59.532 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:19:59.527+0000 7f1fd64e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:19:59.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T17:19:59.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:19:59.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:19:59.924 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:19:59.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:19:59.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:19:59.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:19:59.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:19:59.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:19:59.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:20:00.080 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:20:00.075+0000 7f1fd64e07c0 -1 Falling back to public interface 2024-09-16T17:20:00.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:20:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:20:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:20:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:20:01.359 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:20:01.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:20:01.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:20:01.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:20:02.481 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:20:02.475+0000 7f1fd64e07c0 -1 osd.0 10 log_to_monitors true 2024-09-16T17:20:02.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:20:02.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:20:02.823 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:20:02.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:20:02.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:20:02.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:20:03.307 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 12 up_thru 12 down_at 10 last_clean_interval [5,10) [v2:127.0.0.1:6802/2767754517,v1:127.0.0.1:6803/2767754517] [v2:127.0.0.1:6804/2767754517,v1:127.0.0.1:6805/2767754517] exists,up 68294d40-1585-4437-81ff-1190991a9a32 2024-09-16T17:20:03.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:20:03.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:20:03.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:20:03.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:104: TEST_import_after_merge_and_gap: ceph osd pool set foo pg_num 1 2024-09-16T17:20:04.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:105: TEST_import_after_merge_and_gap: sleep 5 2024-09-16T17:20:09.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: ceph daemon osd.0 perf dump 2024-09-16T17:20:09.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: jq .osd.numpg 2024-09-16T17:20:09.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: grep 2 2024-09-16T17:20:09.243 INFO:tasks.workunit.client.0.smithi039.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-16T17:20:09.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:107: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-16T17:20:09.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:20:09.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:20:09.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:20:09.246 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:20:09.247 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:20:09.247 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:20:09.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:20:09.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:20:09.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:20:09.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:20:09.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:20:09.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:20:09.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:20:09.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:20:09.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:20:09.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T17:20:09.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:20:09.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:20:09.813 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:20:09.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607556 2024-09-16T17:20:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607556 2024-09-16T17:20:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-51539607556' 2024-09-16T17:20:09.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:20:09.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-51539607556 2024-09-16T17:20:09.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:20:09.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:20:09.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-51539607556 2024-09-16T17:20:09.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:20:09.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607556 2024-09-16T17:20:09.990 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 51539607556 2024-09-16T17:20:09.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 51539607556' 2024-09-16T17:20:09.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:20:10.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607554 -lt 51539607556 2024-09-16T17:20:10.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:20:11.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:20:11.393 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:20:11.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607556 -lt 51539607556 2024-09-16T17:20:11.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:20:11.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:20:11.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:20:12.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T17:20:12.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:20:12.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:20:12.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:20:12.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:20:12.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:20:12.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:20:12.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:20:12.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T17:20:12.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:20:12.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:20:12.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:20:13.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T17:20:13.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:20:13.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:20:13.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:110: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-09-16T17:20:13.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:20:13.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:20:13.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:20:13.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:20:13.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:20:13.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:20:13.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:111: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-09-16T17:20:15.046 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-16T17:20:15.046 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-16T17:20:15.046 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-16T17:20:15.689 INFO:tasks.workunit.client.0.smithi039.stdout:Remove successful 2024-09-16T17:20:16.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:113: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-09-16T17:20:16.920 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e20 pg_nums {1={7=2,19=1}} deleted_pools ) 2024-09-16T17:20:16.921 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.1 2024-09-16T17:20:16.921 INFO:tasks.workunit.client.0.smithi039.stdout:pool 1 pg_num 2 -> 1 2024-09-16T17:20:16.921 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81444e03:::benchmark_data_smithi039_455948_object191:head# 2024-09-16T17:20:16.921 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:16.983 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81524550:::benchmark_data_smithi039_455948_object291:head# 2024-09-16T17:20:16.983 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.025 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81688782:::benchmark_data_smithi039_455948_object114:head# 2024-09-16T17:20:17.025 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8340282c:::benchmark_data_smithi039_455948_object261:head# 2024-09-16T17:20:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.108 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:84461d11:::benchmark_data_smithi039_455948_object236:head# 2024-09-16T17:20:17.109 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.158 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:85bbfc69:::benchmark_data_smithi039_455948_object30:head# 2024-09-16T17:20:17.158 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.234 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:861aea5f:::benchmark_data_smithi039_455948_object258:head# 2024-09-16T17:20:17.234 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.276 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:869e8a5e:::benchmark_data_smithi039_455948_object167:head# 2024-09-16T17:20:17.276 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.318 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:872ef1cc:::benchmark_data_smithi039_455948_object195:head# 2024-09-16T17:20:17.318 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.402 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8923ed3b:::benchmark_data_smithi039_455948_object5:head# 2024-09-16T17:20:17.402 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.460 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8a3ec687:::benchmark_data_smithi039_455948_object214:head# 2024-09-16T17:20:17.460 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.519 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8aec744d:::benchmark_data_smithi039_455948_object254:head# 2024-09-16T17:20:17.519 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.561 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8c74c0d6:::benchmark_data_smithi039_455948_object116:head# 2024-09-16T17:20:17.561 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.602 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8eb4657a:::benchmark_data_smithi039_455948_object163:head# 2024-09-16T17:20:17.603 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.644 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92b8125e:::benchmark_data_smithi039_455948_object165:head# 2024-09-16T17:20:17.644 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.686 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92dd79ef:::benchmark_data_smithi039_455948_object130:head# 2024-09-16T17:20:17.686 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.728 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92ec70a0:::benchmark_data_smithi039_455948_object170:head# 2024-09-16T17:20:17.728 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.770 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92f0e148:::benchmark_data_smithi039_455948_object223:head# 2024-09-16T17:20:17.770 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.812 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:930eb7ac:::benchmark_data_smithi039_455948_object262:head# 2024-09-16T17:20:17.812 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.853 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:972245e1:::benchmark_data_smithi039_455948_object7:head# 2024-09-16T17:20:17.853 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.895 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:972cbf49:::benchmark_data_smithi039_455948_object3:head# 2024-09-16T17:20:17.895 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.937 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:984c6070:::benchmark_data_smithi039_455948_object1:head# 2024-09-16T17:20:17.937 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:17.979 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:98946638:::benchmark_data_smithi039_455948_object219:head# 2024-09-16T17:20:17.979 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.020 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:98a59f74:::benchmark_data_smithi039_455948_object216:head# 2024-09-16T17:20:18.020 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.062 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:98b0eb5d:::benchmark_data_smithi039_455948_object131:head# 2024-09-16T17:20:18.062 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.104 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:99654740:::benchmark_data_smithi039_455948_object141:head# 2024-09-16T17:20:18.104 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.146 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9b110e6f:::benchmark_data_smithi039_455948_object55:head# 2024-09-16T17:20:18.146 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.188 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9c86eb42:::benchmark_data_smithi039_455948_object287:head# 2024-09-16T17:20:18.188 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.221 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9c901805:::benchmark_data_smithi039_455948_object59:head# 2024-09-16T17:20:18.221 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.255 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9d050ed9:::benchmark_data_smithi039_455948_object284:head# 2024-09-16T17:20:18.255 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.288 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9e116042:::benchmark_data_smithi039_455948_object286:head# 2024-09-16T17:20:18.288 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.322 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9eac4879:::benchmark_data_smithi039_455948_object289:head# 2024-09-16T17:20:18.322 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.355 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9f0bc29c:::benchmark_data_smithi039_455948_object257:head# 2024-09-16T17:20:18.355 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.389 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a0acf962:::benchmark_data_smithi039_455948_object61:head# 2024-09-16T17:20:18.389 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.422 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a0b1ff33:::benchmark_data_smithi039_455948_object133:head# 2024-09-16T17:20:18.422 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.455 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a299f295:::benchmark_data_smithi039_455948_object176:head# 2024-09-16T17:20:18.455 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.489 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a3610da1:::benchmark_data_smithi039_455948_object149:head# 2024-09-16T17:20:18.489 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.522 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a3616c00:::benchmark_data_smithi039_455948_object246:head# 2024-09-16T17:20:18.522 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.556 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a3d8a56b:::benchmark_data_smithi039_455948_object73:head# 2024-09-16T17:20:18.556 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.589 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a437720d:::benchmark_data_smithi039_455948_object13:head# 2024-09-16T17:20:18.590 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.623 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5276dd6:::benchmark_data_smithi039_455948_object2:head# 2024-09-16T17:20:18.623 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.656 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5e30e12:::benchmark_data_smithi039_455948_object17:head# 2024-09-16T17:20:18.657 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.690 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5e964e3:::benchmark_data_smithi039_455948_object45:head# 2024-09-16T17:20:18.690 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.723 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5f6a828:::benchmark_data_smithi039_455948_object25:head# 2024-09-16T17:20:18.723 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.757 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a63079ff:::benchmark_data_smithi039_455948_object265:head# 2024-09-16T17:20:18.757 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.790 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a64c1b86:::benchmark_data_smithi039_455948_object288:head# 2024-09-16T17:20:18.790 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.823 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a6698c47:::benchmark_data_smithi039_455948_object259:head# 2024-09-16T17:20:18.824 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.857 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a69d0e24:::benchmark_data_smithi039_455948_object113:head# 2024-09-16T17:20:18.857 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.891 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a6c2f07b:::benchmark_data_smithi039_455948_object212:head# 2024-09-16T17:20:18.891 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.924 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a8211e63:::benchmark_data_smithi039_455948_object281:head# 2024-09-16T17:20:18.924 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.957 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ad13fc26:::benchmark_data_smithi039_455948_object110:head# 2024-09-16T17:20:18.958 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:18.991 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ad8239d1:::benchmark_data_smithi039_455948_object232:head# 2024-09-16T17:20:18.991 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.024 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ad91945e:::benchmark_data_smithi039_455948_object93:head# 2024-09-16T17:20:19.024 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.058 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:adaffd31:::benchmark_data_smithi039_455948_object26:head# 2024-09-16T17:20:19.058 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.091 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aeb14177:::benchmark_data_smithi039_455948_object29:head# 2024-09-16T17:20:19.091 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.133 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aecd8d89:::benchmark_data_smithi039_455948_object174:head# 2024-09-16T17:20:19.133 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.209 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b03ef840:::benchmark_data_smithi039_455948_object295:head# 2024-09-16T17:20:19.209 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.259 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b0c8cac8:::benchmark_data_smithi039_455948_object117:head# 2024-09-16T17:20:19.259 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.300 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b137eb01:::benchmark_data_smithi039_455948_object211:head# 2024-09-16T17:20:19.301 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.349 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b2347b11:::benchmark_data_smithi039_455948_object301:head# 2024-09-16T17:20:19.349 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.443 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b249856d:::benchmark_data_smithi039_455948_object235:head# 2024-09-16T17:20:19.443 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.527 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b3253c9c:::benchmark_data_smithi039_455948_object255:head# 2024-09-16T17:20:19.527 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.569 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b5b57515:::benchmark_data_smithi039_455948_object129:head# 2024-09-16T17:20:19.569 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.610 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b6c46095:::benchmark_data_smithi039_455948_object178:head# 2024-09-16T17:20:19.611 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.652 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b6ce9049:::benchmark_data_smithi039_455948_object172:head# 2024-09-16T17:20:19.652 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.702 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b6cf5c8c:::benchmark_data_smithi039_455948_object231:head# 2024-09-16T17:20:19.702 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.744 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b7c8f422:::benchmark_data_smithi039_455948_object112:head# 2024-09-16T17:20:19.744 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.786 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-16T17:20:19.786 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.828 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b8b98acb:::benchmark_data_smithi039_455948_object313:head# 2024-09-16T17:20:19.828 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.870 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b9991d10:::benchmark_data_smithi039_455948_object154:head# 2024-09-16T17:20:19.870 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.912 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:baf26fd7:::benchmark_data_smithi039_455948_object134:head# 2024-09-16T17:20:19.912 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.953 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bb71b590:::benchmark_data_smithi039_455948_object314:head# 2024-09-16T17:20:19.954 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:19.995 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bc052407:::benchmark_data_smithi039_455948_object270:head# 2024-09-16T17:20:19.995 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.037 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bcff6325:::benchmark_data_smithi039_455948_object315:head# 2024-09-16T17:20:20.037 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.078 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bd19ab08:::benchmark_data_smithi039_455948_object81:head# 2024-09-16T17:20:20.078 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.120 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:be7390f5:::benchmark_data_smithi039_455948_object83:head# 2024-09-16T17:20:20.120 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.162 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c07324f3:::benchmark_data_smithi039_455948_object302:head# 2024-09-16T17:20:20.162 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.204 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c0d972fb:::benchmark_data_smithi039_455948_object209:head# 2024-09-16T17:20:20.204 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.246 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c34a47c1:::benchmark_data_smithi039_455948_object319:head# 2024-09-16T17:20:20.246 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.287 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c38689dc:::benchmark_data_smithi039_455948_object50:head# 2024-09-16T17:20:20.287 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.329 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c3893c52:::benchmark_data_smithi039_455948_object66:head# 2024-09-16T17:20:20.329 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.392 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c464c3ce:::benchmark_data_smithi039_455948_object106:head# 2024-09-16T17:20:20.392 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.430 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c56ee08e:::benchmark_data_smithi039_455948_object125:head# 2024-09-16T17:20:20.430 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.471 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c6555459:::benchmark_data_smithi039_455948_object282:head# 2024-09-16T17:20:20.471 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.513 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c70cd8e6:::benchmark_data_smithi039_455948_object181:head# 2024-09-16T17:20:20.513 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.555 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c72b9798:::benchmark_data_smithi039_455948_object186:head# 2024-09-16T17:20:20.555 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.597 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c7547bd4:::benchmark_data_smithi039_455948_object160:head# 2024-09-16T17:20:20.597 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.638 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c8d50a66:::benchmark_data_smithi039_455948_object15:head# 2024-09-16T17:20:20.639 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.680 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c916bf36:::benchmark_data_smithi039_455948_object300:head# 2024-09-16T17:20:20.680 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.714 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c98347ce:::benchmark_data_smithi039_455948_object44:head# 2024-09-16T17:20:20.714 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.747 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ca283ebf:::benchmark_data_smithi039_455948_object245:head# 2024-09-16T17:20:20.747 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.781 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cb190b2d:::benchmark_data_smithi039_455948_object299:head# 2024-09-16T17:20:20.781 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.814 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cb44df0b:::benchmark_data_smithi039_455948_object102:head# 2024-09-16T17:20:20.814 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.848 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cb8dc5ee:::benchmark_data_smithi039_455948_object201:head# 2024-09-16T17:20:20.848 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.881 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cd24ea32:::benchmark_data_smithi039_455948_object42:head# 2024-09-16T17:20:20.881 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.914 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cd94f270:::benchmark_data_smithi039_455948_object124:head# 2024-09-16T17:20:20.915 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.948 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ce974f23:::benchmark_data_smithi039_455948_object159:head# 2024-09-16T17:20:20.948 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:20.981 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cf1dabff:::benchmark_data_smithi039_455948_object151:head# 2024-09-16T17:20:20.981 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.015 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cf56f22e:::benchmark_data_smithi039_455948_object294:head# 2024-09-16T17:20:21.015 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.048 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d04eceaf:::benchmark_data_smithi039_455948_object39:head# 2024-09-16T17:20:21.048 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.082 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d06d3616:::benchmark_data_smithi039_455948_object24:head# 2024-09-16T17:20:21.082 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.115 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d08457ea:::benchmark_data_smithi039_455948_object4:head# 2024-09-16T17:20:21.115 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.157 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d0851a13:::benchmark_data_smithi039_455948_object306:head# 2024-09-16T17:20:21.158 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.216 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d08dfc7d:::benchmark_data_smithi039_455948_object89:head# 2024-09-16T17:20:21.216 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.266 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d1c0a25d:::benchmark_data_smithi039_455948_object105:head# 2024-09-16T17:20:21.266 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.300 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d2347595:::benchmark_data_smithi039_455948_object123:head# 2024-09-16T17:20:21.300 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.333 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d2f20841:::benchmark_data_smithi039_455948_object95:head# 2024-09-16T17:20:21.333 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.375 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d33b721d:::benchmark_data_smithi039_455948_object244:head# 2024-09-16T17:20:21.375 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.442 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d37b01a7:::benchmark_data_smithi039_455948_object243:head# 2024-09-16T17:20:21.442 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.509 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d432d0ae:::benchmark_data_smithi039_455948_object99:head# 2024-09-16T17:20:21.509 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.551 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d4cfa604:::benchmark_data_smithi039_455948_object14:head# 2024-09-16T17:20:21.551 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.593 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d66a0831:::benchmark_data_smithi039_455948_object143:head# 2024-09-16T17:20:21.593 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.635 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d7dff5fe:::benchmark_data_smithi039_455948_object51:head# 2024-09-16T17:20:21.635 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.677 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d81d2707:::benchmark_data_smithi039_455948_object248:head# 2024-09-16T17:20:21.677 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.718 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d8cdeb85:::benchmark_data_smithi039_455948_object279:head# 2024-09-16T17:20:21.718 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.760 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d95fa43b:::benchmark_data_smithi039_455948_object97:head# 2024-09-16T17:20:21.760 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.802 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:db0a03c3:::benchmark_data_smithi039_455948_object35:head# 2024-09-16T17:20:21.802 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.844 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dbe2ac53:::benchmark_data_smithi039_455948_object92:head# 2024-09-16T17:20:21.844 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.886 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e05e2d0e:::benchmark_data_smithi039_455948_object127:head# 2024-09-16T17:20:21.886 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.927 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e18f9e35:::benchmark_data_smithi039_455948_object82:head# 2024-09-16T17:20:21.927 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:21.969 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e1a4e1b7:::benchmark_data_smithi039_455948_object189:head# 2024-09-16T17:20:21.969 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.011 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e2081651:::benchmark_data_smithi039_455948_object303:head# 2024-09-16T17:20:22.011 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.053 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e36479d3:::benchmark_data_smithi039_455948_object277:head# 2024-09-16T17:20:22.053 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.095 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e4509956:::benchmark_data_smithi039_455948_object222:head# 2024-09-16T17:20:22.095 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.136 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e45ad9a8:::benchmark_data_smithi039_455948_object310:head# 2024-09-16T17:20:22.136 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.178 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e49f1f25:::benchmark_data_smithi039_455948_object60:head# 2024-09-16T17:20:22.178 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.220 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e8279365:::benchmark_data_smithi039_455948_object162:head# 2024-09-16T17:20:22.220 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.262 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e84b1af3:::benchmark_data_smithi039_455948_object168:head# 2024-09-16T17:20:22.262 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.303 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e860450e:::benchmark_data_smithi039_455948_object40:head# 2024-09-16T17:20:22.303 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.345 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e9546805:::benchmark_data_smithi039_455948_object84:head# 2024-09-16T17:20:22.345 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.387 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e9c31f48:::benchmark_data_smithi039_455948_object57:head# 2024-09-16T17:20:22.387 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.429 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e9db3ae8:::benchmark_data_smithi039_455948_object185:head# 2024-09-16T17:20:22.429 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.471 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ea00b668:::benchmark_data_smithi039_455948_object126:head# 2024-09-16T17:20:22.471 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.512 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:eac552d0:::benchmark_data_smithi039_455948_object217:head# 2024-09-16T17:20:22.513 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.554 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ed7c17e0:::benchmark_data_smithi039_455948_object152:head# 2024-09-16T17:20:22.554 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.596 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:edc14e53:::benchmark_data_smithi039_455948_object230:head# 2024-09-16T17:20:22.596 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.638 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:efd04b1c:::benchmark_data_smithi039_455948_object225:head# 2024-09-16T17:20:22.638 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.746 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:efea2d35:::benchmark_data_smithi039_455948_object146:head# 2024-09-16T17:20:22.746 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.872 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f1bb07dd:::benchmark_data_smithi039_455948_object298:head# 2024-09-16T17:20:22.872 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.913 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f1e99f50:::benchmark_data_smithi039_455948_object215:head# 2024-09-16T17:20:22.914 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.955 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f1f5e565:::benchmark_data_smithi039_455948_object307:head# 2024-09-16T17:20:22.955 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:22.997 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f20c71ff:::benchmark_data_smithi039_455948_object267:head# 2024-09-16T17:20:22.997 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.039 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f274efb2:::benchmark_data_smithi039_455948_object27:head# 2024-09-16T17:20:23.039 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.080 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f2a8576b:::benchmark_data_smithi039_455948_object54:head# 2024-09-16T17:20:23.081 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.122 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f2f0a09c:::benchmark_data_smithi039_455948_object227:head# 2024-09-16T17:20:23.123 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.174 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f5996306:::benchmark_data_smithi039_455948_object22:head# 2024-09-16T17:20:23.174 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.248 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f61ef65d:::benchmark_data_smithi039_455948_object111:head# 2024-09-16T17:20:23.248 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.306 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f621bc7f:::benchmark_data_smithi039_455948_object182:head# 2024-09-16T17:20:23.307 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.340 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f72c3135:::benchmark_data_smithi039_455948_object21:head# 2024-09-16T17:20:23.340 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.373 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f7b30eee:::benchmark_data_smithi039_455948_object194:head# 2024-09-16T17:20:23.373 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.449 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f97e448b:::benchmark_data_smithi039_455948_object10:head# 2024-09-16T17:20:23.449 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.507 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f9d8cd97:::benchmark_data_smithi039_455948_object199:head# 2024-09-16T17:20:23.507 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.583 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f9e3593c:::benchmark_data_smithi039_455948_object256:head# 2024-09-16T17:20:23.583 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.616 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fa9bf67e:::benchmark_data_smithi039_455948_object32:head# 2024-09-16T17:20:23.616 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.650 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fadac265:::benchmark_data_smithi039_455948_object305:head# 2024-09-16T17:20:23.650 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.683 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fadc5b0b:::benchmark_data_smithi039_455948_object104:head# 2024-09-16T17:20:23.683 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.717 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fd05df8b:::benchmark_data_smithi039_455948_object70:head# 2024-09-16T17:20:23.717 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.750 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fdea465d:::benchmark_data_smithi039_455948_object107:head# 2024-09-16T17:20:23.750 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.783 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fe001570:::benchmark_data_smithi039_455948_object171:head# 2024-09-16T17:20:23.784 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.817 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ff9caaec:::benchmark_data_smithi039_455948_object263:head# 2024-09-16T17:20:23.817 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:23.851 INFO:tasks.workunit.client.0.smithi039.stdout:write_pg epoch 9 info 1.1( v 9'160 (0'0,9'160] local-lis/les=7/8 n=160 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-16T17:20:23.892 INFO:tasks.workunit.client.0.smithi039.stdout:Import successful 2024-09-16T17:20:24.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:114: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-16T17:20:25.736 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e20 pg_nums {1={7=2,19=1}} deleted_pools ) 2024-09-16T17:20:25.736 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.0 2024-09-16T17:20:25.736 INFO:tasks.workunit.client.0.smithi039.stdout:pool 1 pg_num 2 -> 1 2024-09-16T17:20:25.737 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:00456576:::benchmark_data_smithi039_455948_object157:head# 2024-09-16T17:20:25.737 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:25.815 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:014ebcb1:::benchmark_data_smithi039_455948_object233:head# 2024-09-16T17:20:25.815 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:25.856 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:01dc36eb:::benchmark_data_smithi039_455948_object64:head# 2024-09-16T17:20:25.857 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:25.898 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:01f0fd22:::benchmark_data_smithi039_455948_object208:head# 2024-09-16T17:20:25.898 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:25.940 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:01f5a6f9:::benchmark_data_smithi039_455948_object272:head# 2024-09-16T17:20:25.940 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:25.982 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:02207eee:::benchmark_data_smithi039_455948_object192:head# 2024-09-16T17:20:25.982 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.015 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:02bddb95:::benchmark_data_smithi039_455948_object137:head# 2024-09-16T17:20:26.015 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.049 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:033989ba:::benchmark_data_smithi039_455948_object80:head# 2024-09-16T17:20:26.049 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.082 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:041f4598:::benchmark_data_smithi039_455948_object187:head# 2024-09-16T17:20:26.082 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.116 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:047a0c2d:::benchmark_data_smithi039_455948_object293:head# 2024-09-16T17:20:26.116 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.149 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:04aae244:::benchmark_data_smithi039_455948_object98:head# 2024-09-16T17:20:26.149 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.183 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:058f836c:::benchmark_data_smithi039_455948_object205:head# 2024-09-16T17:20:26.183 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.216 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:05b9dc7f:::benchmark_data_smithi039_455948_object184:head# 2024-09-16T17:20:26.216 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.249 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:05f32019:::benchmark_data_smithi039_455948_object180:head# 2024-09-16T17:20:26.250 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.283 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0a2f45e2:::benchmark_data_smithi039_455948_object136:head# 2024-09-16T17:20:26.283 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.316 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0a3ddd05:::benchmark_data_smithi039_455948_object229:head# 2024-09-16T17:20:26.317 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.358 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0dc71e56:::benchmark_data_smithi039_455948_object224:head# 2024-09-16T17:20:26.358 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.392 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0e79e577:::benchmark_data_smithi039_455948_object308:head# 2024-09-16T17:20:26.392 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.425 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0e80050a:::benchmark_data_smithi039_455948_object9:head# 2024-09-16T17:20:26.425 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.459 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0f01eefd:::benchmark_data_smithi039_455948_object166:head# 2024-09-16T17:20:26.459 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.492 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0fcc09dc:::benchmark_data_smithi039_455948_object175:head# 2024-09-16T17:20:26.492 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.526 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:100e9870:::benchmark_data_smithi039_455948_object122:head# 2024-09-16T17:20:26.526 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.559 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:12727e65:::benchmark_data_smithi039_455948_object164:head# 2024-09-16T17:20:26.559 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.593 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:12810ad7:::benchmark_data_smithi039_455948_object132:head# 2024-09-16T17:20:26.593 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.626 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:12cae338:::benchmark_data_smithi039_455948_object213:head# 2024-09-16T17:20:26.626 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.660 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:13b7075e:::benchmark_data_smithi039_455948_object292:head# 2024-09-16T17:20:26.660 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.693 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:14072aff:::benchmark_data_smithi039_455948_object266:head# 2024-09-16T17:20:26.693 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.727 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:14a5c78e:::benchmark_data_smithi039_455948_object46:head# 2024-09-16T17:20:26.727 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.760 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1561ff76:::benchmark_data_smithi039_455948_object155:head# 2024-09-16T17:20:26.760 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.794 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:16d84de3:::benchmark_data_smithi039_455948_object200:head# 2024-09-16T17:20:26.794 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.827 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1a3cf39c:::benchmark_data_smithi039_455948_object177:head# 2024-09-16T17:20:26.827 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.860 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1ae97a14:::benchmark_data_smithi039_455948_object69:head# 2024-09-16T17:20:26.861 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.894 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1df9e5c8:::benchmark_data_smithi039_455948_object115:head# 2024-09-16T17:20:26.894 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.927 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1e6aee90:::benchmark_data_smithi039_455948_object312:head# 2024-09-16T17:20:26.927 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:26.978 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1e9dda24:::benchmark_data_smithi039_455948_object173:head# 2024-09-16T17:20:26.978 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.011 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1ea64c7e:::benchmark_data_smithi039_455948_object278:head# 2024-09-16T17:20:27.011 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.044 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1fb0f211:::benchmark_data_smithi039_455948_object109:head# 2024-09-16T17:20:27.045 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.078 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1fbf7b6c:::benchmark_data_smithi039_455948_object207:head# 2024-09-16T17:20:27.078 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.111 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1fd4abd6:::benchmark_data_smithi039_455948_object142:head# 2024-09-16T17:20:27.111 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.145 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:20997167:::benchmark_data_smithi039_455948_object56:head# 2024-09-16T17:20:27.145 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.228 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:227f4aa5:::benchmark_data_smithi039_455948_object260:head# 2024-09-16T17:20:27.229 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.279 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:241f88ec:::benchmark_data_smithi039_455948_object118:head# 2024-09-16T17:20:27.279 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.312 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2464192d:::benchmark_data_smithi039_455948_object67:head# 2024-09-16T17:20:27.312 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.346 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:255d9356:::benchmark_data_smithi039_455948_object19:head# 2024-09-16T17:20:27.346 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.379 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:260b2bbc:::benchmark_data_smithi039_455948_object79:head# 2024-09-16T17:20:27.379 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.464 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:27914c55:::benchmark_data_smithi039_455948_object135:head# 2024-09-16T17:20:27.464 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.547 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:279e6bbf:::benchmark_data_smithi039_455948_object247:head# 2024-09-16T17:20:27.547 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.580 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:28d56bea:::benchmark_data_smithi039_455948_object11:head# 2024-09-16T17:20:27.580 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.614 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:294aa246:::benchmark_data_smithi039_455948_object38:head# 2024-09-16T17:20:27.614 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.647 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2a8d03cc:::benchmark_data_smithi039_455948_object197:head# 2024-09-16T17:20:27.647 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.680 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2cdc5a61:::benchmark_data_smithi039_455948_object87:head# 2024-09-16T17:20:27.681 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.714 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2f2af160:::benchmark_data_smithi039_455948_object63:head# 2024-09-16T17:20:27.714 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.747 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2f6ad66d:::benchmark_data_smithi039_455948_object237:head# 2024-09-16T17:20:27.748 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.781 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2fa31139:::benchmark_data_smithi039_455948_object36:head# 2024-09-16T17:20:27.781 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.823 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2faa5036:::benchmark_data_smithi039_455948_object43:head# 2024-09-16T17:20:27.823 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.856 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3061c169:::benchmark_data_smithi039_455948_object271:head# 2024-09-16T17:20:27.856 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.890 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:30da3c6c:::benchmark_data_smithi039_455948_object193:head# 2024-09-16T17:20:27.890 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.923 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3105728e:::benchmark_data_smithi039_455948_object148:head# 2024-09-16T17:20:27.923 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.957 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3173dfe0:::benchmark_data_smithi039_455948_object188:head# 2024-09-16T17:20:27.957 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:27.990 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:335c1a68:::benchmark_data_smithi039_455948_object128:head# 2024-09-16T17:20:27.990 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.032 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:351c3af9:::benchmark_data_smithi039_455948_object72:head# 2024-09-16T17:20:28.032 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.074 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:36bb23c0:::benchmark_data_smithi039_455948_object101:head# 2024-09-16T17:20:28.074 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.116 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:375cb5d3:::benchmark_data_smithi039_455948_object275:head# 2024-09-16T17:20:28.116 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.157 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:37a5f014:::benchmark_data_smithi039_455948_object204:head# 2024-09-16T17:20:28.157 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.199 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3a6cafe9:::benchmark_data_smithi039_455948_object6:head# 2024-09-16T17:20:28.199 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.283 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3c81aa17:::benchmark_data_smithi039_455948_object280:head# 2024-09-16T17:20:28.283 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.316 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3cd1f324:::benchmark_data_smithi039_455948_object202:head# 2024-09-16T17:20:28.316 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.350 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3eaccc36:::benchmark_data_smithi039_455948_object144:head# 2024-09-16T17:20:28.350 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.383 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3f37ec22:::benchmark_data_smithi039_455948_object161:head# 2024-09-16T17:20:28.383 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.417 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3fc02a40:::benchmark_data_smithi039_455948_object74:head# 2024-09-16T17:20:28.417 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.450 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4003301f:::benchmark_data_smithi039_455948_object139:head# 2024-09-16T17:20:28.450 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.484 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4117f431:::benchmark_data_smithi039_455948_object234:head# 2024-09-16T17:20:28.484 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.517 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:41facc71:::benchmark_data_smithi039_455948_object198:head# 2024-09-16T17:20:28.517 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.559 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4411f440:::benchmark_data_smithi039_455948_object297:head# 2024-09-16T17:20:28.559 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.593 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:44802c6d:::benchmark_data_smithi039_455948_object119:head# 2024-09-16T17:20:28.596 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.626 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:45c521fb:::benchmark_data_smithi039_455948_object249:head# 2024-09-16T17:20:28.626 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.660 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:469ace3a:::benchmark_data_smithi039_455948_object206:head# 2024-09-16T17:20:28.660 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.693 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:47970e62:::benchmark_data_smithi039_455948_object210:head# 2024-09-16T17:20:28.693 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.726 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:47b13862:::benchmark_data_smithi039_455948_object138:head# 2024-09-16T17:20:28.727 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.760 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4801d23d:::benchmark_data_smithi039_455948_object85:head# 2024-09-16T17:20:28.760 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.793 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:491895da:::benchmark_data_smithi039_455948_object169:head# 2024-09-16T17:20:28.793 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.827 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4935d551:::benchmark_data_smithi039_455948_object251:head# 2024-09-16T17:20:28.827 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.860 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:496b18ad:::benchmark_data_smithi039_455948_object253:head# 2024-09-16T17:20:28.860 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.894 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4a184ab0:::benchmark_data_smithi039_455948_object158:head# 2024-09-16T17:20:28.894 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.927 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4a1d7637:::benchmark_data_smithi039_455948_object183:head# 2024-09-16T17:20:28.927 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.961 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4ac873c7:::benchmark_data_smithi039_455948_object37:head# 2024-09-16T17:20:28.961 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:28.994 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4bc4a2e0:::benchmark_data_smithi039_455948_object49:head# 2024-09-16T17:20:28.994 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.028 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4c5a7992:::benchmark_data_smithi039_455948_object221:head# 2024-09-16T17:20:29.028 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.061 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4f03ab2e:::benchmark_data_smithi039_455948_object318:head# 2024-09-16T17:20:29.061 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.128 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4f301666:::benchmark_data_smithi039_455948_object228:head# 2024-09-16T17:20:29.128 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.203 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4f3a0a7a:::benchmark_data_smithi039_455948_object240:head# 2024-09-16T17:20:29.203 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.253 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4fb8888a:::benchmark_data_smithi039_455948_object203:head# 2024-09-16T17:20:29.254 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.287 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:50a96376:::benchmark_data_smithi039_455948_object96:head# 2024-09-16T17:20:29.287 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.320 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5244e47b:::benchmark_data_smithi039_455948_object62:head# 2024-09-16T17:20:29.320 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.354 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:52d42a8e:::benchmark_data_smithi039_455948_object48:head# 2024-09-16T17:20:29.354 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.431 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:52ede5be:::benchmark_data_smithi039_455948_object12:head# 2024-09-16T17:20:29.431 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.505 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53a105b2:::benchmark_data_smithi039_455948_object311:head# 2024-09-16T17:20:29.505 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.538 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53cb9fd2:::benchmark_data_smithi039_455948_object190:head# 2024-09-16T17:20:29.538 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.572 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53ed147c:::benchmark_data_smithi039_455948_object31:head# 2024-09-16T17:20:29.572 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.605 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:54d49a7f:::benchmark_data_smithi039_455948_object28:head# 2024-09-16T17:20:29.605 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.639 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:54e04d99:::benchmark_data_smithi039_455948_object283:head# 2024-09-16T17:20:29.639 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.672 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:54e85f22:::benchmark_data_smithi039_455948_object23:head# 2024-09-16T17:20:29.672 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.706 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5651eea5:::benchmark_data_smithi039_455948_object18:head# 2024-09-16T17:20:29.706 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.739 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:57dabe70:::benchmark_data_smithi039_455948_object218:head# 2024-09-16T17:20:29.739 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.772 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5801fe81:::benchmark_data_smithi039_455948_object75:head# 2024-09-16T17:20:29.773 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.806 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:58d2c70a:::benchmark_data_smithi039_455948_object0:head# 2024-09-16T17:20:29.806 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.840 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:59614fc8:::benchmark_data_smithi039_455948_object76:head# 2024-09-16T17:20:29.840 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.873 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5a3d91e5:::benchmark_data_smithi039_455948_object108:head# 2024-09-16T17:20:29.873 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.906 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5a7ae6c2:::benchmark_data_smithi039_455948_object100:head# 2024-09-16T17:20:29.906 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.940 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5c28febb:::benchmark_data_smithi039_455948_object156:head# 2024-09-16T17:20:29.940 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:29.973 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5db907e9:::benchmark_data_smithi039_455948_object121:head# 2024-09-16T17:20:29.974 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.007 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ded13e4:::benchmark_data_smithi039_455948_object90:head# 2024-09-16T17:20:30.007 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.040 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5e428851:::benchmark_data_smithi039_455948_object309:head# 2024-09-16T17:20:30.040 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.074 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5e629290:::benchmark_data_smithi039_455948_object20:head# 2024-09-16T17:20:30.074 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.107 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5efeb4ae:::benchmark_data_smithi039_455948_object153:head# 2024-09-16T17:20:30.107 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.141 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ff59d48:::benchmark_data_smithi039_455948_object16:head# 2024-09-16T17:20:30.141 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.174 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6137bcba:::benchmark_data_smithi039_455948_object120:head# 2024-09-16T17:20:30.174 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.208 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6154b590:::benchmark_data_smithi039_455948_object58:head# 2024-09-16T17:20:30.208 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.241 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6280b741:::benchmark_data_smithi039_455948_object250:head# 2024-09-16T17:20:30.241 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.274 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:62961663:::benchmark_data_smithi039_455948_object296:head# 2024-09-16T17:20:30.274 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.308 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:642ab5c2:::benchmark_data_smithi039_455948_object41:head# 2024-09-16T17:20:30.308 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.350 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:64aa8ba8:::benchmark_data_smithi039_455948_object71:head# 2024-09-16T17:20:30.350 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.383 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:654ddc28:::benchmark_data_smithi039_455948_object147:head# 2024-09-16T17:20:30.383 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.425 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:656a01f0:::benchmark_data_smithi039_455948_object316:head# 2024-09-16T17:20:30.425 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.467 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6667fb4d:::benchmark_data_smithi039_455948_object252:head# 2024-09-16T17:20:30.467 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.509 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:66719837:::benchmark_data_smithi039_455948_object179:head# 2024-09-16T17:20:30.509 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.551 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:67fc60d1:::benchmark_data_smithi039_455948_object103:head# 2024-09-16T17:20:30.551 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.584 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:67fc91c1:::benchmark_data_smithi039_455948_object52:head# 2024-09-16T17:20:30.584 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.618 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6815ad2e:::benchmark_data_smithi039_455948_object196:head# 2024-09-16T17:20:30.618 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.693 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:69293e2b:::benchmark_data_smithi039_455948_object290:head# 2024-09-16T17:20:30.693 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.726 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6ae4f661:::benchmark_data_smithi039_455948_object241:head# 2024-09-16T17:20:30.727 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.760 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6cb11b42:::benchmark_data_smithi039_455948_object285:head# 2024-09-16T17:20:30.760 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.793 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6ce0dbf0:::benchmark_data_smithi039_455948_object268:head# 2024-09-16T17:20:30.793 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.827 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6d2fe8c5:::benchmark_data_smithi039_455948_object273:head# 2024-09-16T17:20:30.827 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.860 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6d540a4e:::benchmark_data_smithi039_455948_object8:head# 2024-09-16T17:20:30.860 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.902 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6d724163:::benchmark_data_smithi039_455948_object269:head# 2024-09-16T17:20:30.902 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.935 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6e61b9d5:::benchmark_data_smithi039_455948_object68:head# 2024-09-16T17:20:30.935 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:30.969 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6f095114:::benchmark_data_smithi039_455948_object94:head# 2024-09-16T17:20:30.969 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.002 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:70691fc8:::benchmark_data_smithi039_455948_object145:head# 2024-09-16T17:20:31.002 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.036 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:708034b3:::benchmark_data_smithi039_455948_object276:head# 2024-09-16T17:20:31.036 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.069 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:70f55f31:::benchmark_data_smithi039_455948_object239:head# 2024-09-16T17:20:31.069 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.103 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:71b1cb82:::benchmark_data_smithi039_455948_object65:head# 2024-09-16T17:20:31.103 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.136 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:71ce8906:::benchmark_data_smithi039_455948_object78:head# 2024-09-16T17:20:31.136 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.212 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:72fdddd5:::benchmark_data_smithi039_455948_object86:head# 2024-09-16T17:20:31.212 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.262 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:73b633f2:::benchmark_data_smithi039_455948_object150:head# 2024-09-16T17:20:31.262 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.295 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:754b4658:::benchmark_data_smithi039_455948_object77:head# 2024-09-16T17:20:31.296 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.329 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:75a63f9f:::benchmark_data_smithi039_455948_object226:head# 2024-09-16T17:20:31.329 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.362 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:75f9bf67:::benchmark_data_smithi039_455948_object242:head# 2024-09-16T17:20:31.362 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.439 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:764ab525:::benchmark_data_smithi039_455948_object317:head# 2024-09-16T17:20:31.439 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.505 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:768e2761:::benchmark_data_smithi039_455948_object238:head# 2024-09-16T17:20:31.505 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.538 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:77182e0f:::benchmark_data_smithi039_455948_object140:head# 2024-09-16T17:20:31.538 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.572 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7739f249:::benchmark_data_smithi039_455948_object34:head# 2024-09-16T17:20:31.572 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.605 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:784592ec:::benchmark_data_smithi039_455948_object33:head# 2024-09-16T17:20:31.605 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.639 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7a597a33:::benchmark_data_smithi039_455948_object274:head# 2024-09-16T17:20:31.639 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.672 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7b291d50:::benchmark_data_smithi039_455948_object220:head# 2024-09-16T17:20:31.672 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.705 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7c07aaad:::benchmark_data_smithi039_455948_object91:head# 2024-09-16T17:20:31.706 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.739 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7cd2d272:::benchmark_data_smithi039_455948_object53:head# 2024-09-16T17:20:31.739 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.772 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7d0d8840:::benchmark_data_smithi039_455948_object264:head# 2024-09-16T17:20:31.773 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.806 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7d342ae1:::benchmark_data_smithi039_455948_object47:head# 2024-09-16T17:20:31.806 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.839 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7ee172f3:::benchmark_data_smithi039_455948_object304:head# 2024-09-16T17:20:31.839 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.873 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7fe7c24d:::benchmark_data_smithi039_455948_object88:head# 2024-09-16T17:20:31.873 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:20:31.907 INFO:tasks.workunit.client.0.smithi039.stdout:write_pg epoch 9 info 1.0( v 9'161 (0'0,9'161] local-lis/les=7/8 n=161 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-16T17:20:31.948 INFO:tasks.workunit.client.0.smithi039.stdout:Import successful 2024-09-16T17:20:32.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:115: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-09-16T17:20:32.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-16T17:20:32.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T17:20:32.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T17:20:32.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T17:20:32.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-16T17:20:32.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:20:32.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:20:32.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:20:32.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:20:32.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-16T17:20:32.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-16T17:20:32.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T17:20:32.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T17:20:32.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-16T17:20:32.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T17:20:32.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:20:32.490 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:20:32.490 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:20:32.490 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:20:32.491 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:20:32.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:20:32.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:20:32.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:20:32.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-16T17:20:32.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-16T17:20:32.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:20:32.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:20:32.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:20:32.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:20:32.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T17:20:32.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T17:20:32.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-16T17:20:32.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T17:20:32.492 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:20:32.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:20:32.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-16T17:20:32.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T17:20:32.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T17:20:32.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T17:20:32.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T17:20:32.526 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:20:32.523+0000 7f4a826c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:20:32.529 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:20:32.527+0000 7f4a826c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:20:32.531 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:20:32.531+0000 7f4a826c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:20:32.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T17:20:32.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:20:32.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:20:32.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:20:32.921 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:20:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:20:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:20:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:20:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:20:32.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:20:33.093 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:20:33.091+0000 7f4a826c27c0 -1 Falling back to public interface 2024-09-16T17:20:33.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:20:34.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:20:34.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:20:34.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:20:34.347 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:20:34.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:20:34.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:20:34.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:20:35.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:20:35.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:20:35.779 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:20:35.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:20:35.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:20:35.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:20:36.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:20:36.396 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:20:36.395+0000 7f4a826c27c0 -1 osd.0 20 log_to_monitors true 2024-09-16T17:20:37.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:20:37.219 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:20:37.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:20:37.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:20:37.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:20:37.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:20:37.852 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 21 last_clean_interval [12,20) [v2:127.0.0.1:6802/3925174340,v1:127.0.0.1:6803/3925174340] [v2:127.0.0.1:6804/3925174340,v1:127.0.0.1:6805/3925174340] exists,up 68294d40-1585-4437-81ff-1190991a9a32 2024-09-16T17:20:37.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:20:37.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:20:37.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:20:37.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:117: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-16T17:20:37.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:20:37.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:20:37.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:20:37.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:20:37.854 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:20:37.854 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:20:37.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:20:37.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:20:37.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:20:37.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:20:37.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:20:37.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:20:37.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:20:37.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:20:37.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:20:38.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T17:20:38.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:20:38.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:20:38.405 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:20:38.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=98784247811 2024-09-16T17:20:38.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 98784247811 2024-09-16T17:20:38.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-98784247811' 2024-09-16T17:20:38.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:20:38.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-98784247811 2024-09-16T17:20:38.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:20:38.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:20:38.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-98784247811 2024-09-16T17:20:38.586 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:20:38.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=98784247811 2024-09-16T17:20:38.588 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 98784247811 2024-09-16T17:20:38.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 98784247811' 2024-09-16T17:20:38.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:20:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 98784247811 2024-09-16T17:20:39.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:20:40.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:20:40.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:20:40.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 98784247811 -lt 98784247811 2024-09-16T17:20:40.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:20:40.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:20:40.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:20:40.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T17:20:40.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:20:40.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:20:40.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:20:40.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:20:40.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:20:40.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:20:40.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:20:41.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T17:20:41.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:20:41.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:20:41.401 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:20:41.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T17:20:41.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:20:41.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:20:41.939 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: seq 1 50 2024-09-16T17:20:41.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:20:41.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:20:42.747 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:20:42.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:20:43.534 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:20:43.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:20:43.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:20:44.674 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:20:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:20:45.459 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:20:45.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:20:45.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:20:46.288 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:20:46.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:20:47.259 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:20:47.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:20:47.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:20:48.053 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:20:48.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:20:48.856 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:20:48.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:20:48.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:20:49.660 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:20:49.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:20:50.464 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:20:50.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:20:50.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:20:51.447 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:20:51.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:20:52.205 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:20:52.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:20:52.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:20:53.025 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:20:53.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:20:53.894 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:20:53.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:20:53.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:20:54.715 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:20:54.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:20:55.635 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:20:55.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:20:55.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:20:56.482 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:20:56.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:20:57.428 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:20:57.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:20:57.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:20:58.248 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:20:58.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:20:59.042 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:20:59.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:20:59.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:20:59.811 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:20:59.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:00.623 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:00.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:00.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:01.603 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:01.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:02.346 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:02.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:02.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:03.133 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:03.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:04.019 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:04.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:04.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:05.041 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:05.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:05.920 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:05.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:05.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:06.739 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:06.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:07.651 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:07.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:07.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:08.436 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:08.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:09.396 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:09.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:09.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:10.135 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:10.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:10.979 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:11.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:11.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:11.867 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:11.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:12.678 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:12.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:12.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:13.548 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:13.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:14.242 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:14.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:14.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:15.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:15.806 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:15.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:15.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:16.576 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:16.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:17.573 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:17.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:17.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:18.358 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:18.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:19.219 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:19.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:19.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:20.073 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:20.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:20.810 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:20.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:20.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:21.688 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:21.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:22.508 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:22.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:22.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:23.303 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:23.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:24.106 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:24.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:24.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:25.160 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:25.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:25.971 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:25.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:25.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:26.716 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:26.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:27.687 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:27.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:27.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:28.456 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:28.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:29.443 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:29.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:29.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:30.178 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:30.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:31.040 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:31.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:31.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:31.969 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:31.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:32.772 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:32.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:32.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:33.768 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:33.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:34.537 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:34.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:34.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:35.508 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:35.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:36.277 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:36.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:36.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:37.156 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:37.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:38.059 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:38.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:38.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:38.812 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:38.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:39.657 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:39.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:39.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:40.519 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:40.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:41.414 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:41.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:41.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:42.233 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:42.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:43.046 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:43.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:43.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:43.932 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:43.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:44.854 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:44.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:44.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:45.648 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:45.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:46.535 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:46.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:46.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:47.383 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:47.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:48.189 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:48.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:48.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:49.085 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:49.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:49.880 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:49.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:49.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:50.775 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:50.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:51.678 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:51.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:51.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:52.439 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:52.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:53.369 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:53.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:53.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:54.204 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:54.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:55.324 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:55.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:55.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:56.151 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:56.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:57.088 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:57.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:57.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:57.888 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:57.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:21:58.775 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:21:58.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:21:58.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:21:59.578 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:21:59.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:22:00.414 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:22:00.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:22:00.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:22:01.528 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:22:01.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:22:02.414 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:22:02.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:22:02.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:22:03.340 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:22:03.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:22:04.213 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:22:04.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:22:04.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:22:05.174 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:22:05.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:22:05.953 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:22:05.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-16T17:22:05.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:22:06.856 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:22:06.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:22:07.701 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:22:07.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:127: TEST_import_after_merge_and_gap: rados -p foo bench 1 write -b 1024 --no-cleanup 2024-09-16T17:22:07.766 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-16T17:22:07.766 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 1 seconds or 0 objects 2024-09-16T17:22:07.767 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_461800 2024-09-16T17:22:07.767 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T17:22:07.767 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-16T17:22:08.767 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 177 161 0.157211 0.157227 0.0750389 0.0940494 2024-09-16T17:22:09.767 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 1.05243 2024-09-16T17:22:09.767 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 177 2024-09-16T17:22:09.767 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 1024 2024-09-16T17:22:09.767 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 1024 2024-09-16T17:22:09.767 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.16424 2024-09-16T17:22:09.767 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0 2024-09-16T17:22:09.767 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.157227 2024-09-16T17:22:09.768 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.157227 2024-09-16T17:22:09.768 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 168 2024-09-16T17:22:09.768 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 0 2024-09-16T17:22:09.768 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 161 2024-09-16T17:22:09.768 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 161 2024-09-16T17:22:09.768 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.0923449 2024-09-16T17:22:09.768 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.0198163 2024-09-16T17:22:09.768 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 0.142788 2024-09-16T17:22:09.768 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.0735522 2024-09-16T17:22:09.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:128: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-16T17:22:09.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:22:09.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:22:09.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:22:09.810 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:22:09.811 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:22:09.811 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:22:09.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:22:09.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:22:09.812 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:22:09.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:22:09.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:22:09.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:22:09.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:22:09.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:22:09.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:22:10.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T17:22:10.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:22:10.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:22:10.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:22:10.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=98784247831 2024-09-16T17:22:10.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 98784247831 2024-09-16T17:22:10.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-98784247831' 2024-09-16T17:22:10.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:22:10.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-98784247831 2024-09-16T17:22:10.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:22:10.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:22:10.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-98784247831 2024-09-16T17:22:10.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:22:10.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=98784247831 2024-09-16T17:22:10.576 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 98784247831 2024-09-16T17:22:10.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 98784247831' 2024-09-16T17:22:10.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:22:10.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 98784247829 -lt 98784247831 2024-09-16T17:22:10.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:22:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:22:11.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:22:12.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 98784247831 -lt 98784247831 2024-09-16T17:22:12.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:22:12.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:22:12.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:22:12.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T17:22:12.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:22:12.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:22:12.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:22:12.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:22:12.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:22:12.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:22:12.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:22:13.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T17:22:13.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:22:13.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:22:13.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:22:13.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T17:22:13.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:22:13.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:22:13.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:129: TEST_import_after_merge_and_gap: ceph tell osd.0 send_beacon 2024-09-16T17:22:14.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:130: TEST_import_after_merge_and_gap: sleep 5 2024-09-16T17:22:19.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:131: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-16T17:22:19.971 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-16T17:22:19.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:132: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-16T17:22:20.782 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is unset 2024-09-16T17:22:20.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:133: TEST_import_after_merge_and_gap: sleep 5 2024-09-16T17:22:25.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:135: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-09-16T17:22:25.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:22:25.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:22:25.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:22:25.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:22:25.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:22:26.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:22:26.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:138: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-09-16T17:22:27.139 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-16T17:22:27.140 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.1 2024-09-16T17:22:27.140 INFO:tasks.workunit.client.0.smithi039.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-16T17:22:27.140 INFO:tasks.workunit.client.0.smithi039.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-09-16T17:22:27.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:140: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export-remove --pgid 1.0 --force --file td/pg-split-merge/1.0.later 2024-09-16T17:22:28.435 INFO:tasks.workunit.client.0.smithi039.stderr:Exporting 1.0 info 1.0( v 224'339 (9'161,224'339] local-lis/les=23/24 n=498 ec=7/7 lis/c=23/23 les/c/f=24/24/0 sis=23) 2024-09-16T17:22:28.456 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:00456576:::benchmark_data_smithi039_455948_object157:head# 2024-09-16T17:22:28.465 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:00eb4400:::benchmark_data_smithi039_461800_object119:head# 2024-09-16T17:22:28.466 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:014ebcb1:::benchmark_data_smithi039_455948_object233:head# 2024-09-16T17:22:28.467 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:01dc36eb:::benchmark_data_smithi039_455948_object64:head# 2024-09-16T17:22:28.467 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:01f0fd22:::benchmark_data_smithi039_455948_object208:head# 2024-09-16T17:22:28.468 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:01f5a6f9:::benchmark_data_smithi039_455948_object272:head# 2024-09-16T17:22:28.470 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:02207eee:::benchmark_data_smithi039_455948_object192:head# 2024-09-16T17:22:28.470 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:02bddb95:::benchmark_data_smithi039_455948_object137:head# 2024-09-16T17:22:28.470 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:033989ba:::benchmark_data_smithi039_455948_object80:head# 2024-09-16T17:22:28.470 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:041f4598:::benchmark_data_smithi039_455948_object187:head# 2024-09-16T17:22:28.470 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:047a0c2d:::benchmark_data_smithi039_455948_object293:head# 2024-09-16T17:22:28.470 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:048eb4a3:::benchmark_data_smithi039_461800_object10:head# 2024-09-16T17:22:28.471 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:04aae244:::benchmark_data_smithi039_455948_object98:head# 2024-09-16T17:22:28.471 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:058f836c:::benchmark_data_smithi039_455948_object205:head# 2024-09-16T17:22:28.472 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:05b9dc7f:::benchmark_data_smithi039_455948_object184:head# 2024-09-16T17:22:28.472 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:05f32019:::benchmark_data_smithi039_455948_object180:head# 2024-09-16T17:22:28.473 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:06327336:::benchmark_data_smithi039_461800_object122:head# 2024-09-16T17:22:28.473 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:064aeecb:::benchmark_data_smithi039_461800_object158:head# 2024-09-16T17:22:28.474 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0681ed1a:::benchmark_data_smithi039_461800_object170:head# 2024-09-16T17:22:28.474 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0749227f:::benchmark_data_smithi039_461800_object50:head# 2024-09-16T17:22:28.475 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:07512169:::benchmark_data_smithi039_461800_object77:head# 2024-09-16T17:22:28.475 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:07fcaf55:::benchmark_data_smithi039_461800_object45:head# 2024-09-16T17:22:28.475 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0a2f45e2:::benchmark_data_smithi039_455948_object136:head# 2024-09-16T17:22:28.476 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0a3ddd05:::benchmark_data_smithi039_455948_object229:head# 2024-09-16T17:22:28.476 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0d9d2be8:::benchmark_data_smithi039_461800_object101:head# 2024-09-16T17:22:28.477 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0dc71e56:::benchmark_data_smithi039_455948_object224:head# 2024-09-16T17:22:28.477 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0e128744:::benchmark_data_smithi039_461800_object42:head# 2024-09-16T17:22:28.478 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0e79e577:::benchmark_data_smithi039_455948_object308:head# 2024-09-16T17:22:28.478 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0e80050a:::benchmark_data_smithi039_455948_object9:head# 2024-09-16T17:22:28.478 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0f01eefd:::benchmark_data_smithi039_455948_object166:head# 2024-09-16T17:22:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0fcc09dc:::benchmark_data_smithi039_455948_object175:head# 2024-09-16T17:22:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:100e9870:::benchmark_data_smithi039_455948_object122:head# 2024-09-16T17:22:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1060baad:::benchmark_data_smithi039_461800_object113:head# 2024-09-16T17:22:28.483 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1074fdbd:::benchmark_data_smithi039_461800_object115:head# 2024-09-16T17:22:28.484 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:12716124:::benchmark_data_smithi039_461800_object85:head# 2024-09-16T17:22:28.485 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:12727e65:::benchmark_data_smithi039_455948_object164:head# 2024-09-16T17:22:28.485 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:12810ad7:::benchmark_data_smithi039_455948_object132:head# 2024-09-16T17:22:28.485 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:12cae338:::benchmark_data_smithi039_455948_object213:head# 2024-09-16T17:22:28.486 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1330884b:::benchmark_data_smithi039_461800_object68:head# 2024-09-16T17:22:28.487 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:138e2f90:::benchmark_data_smithi039_461800_object43:head# 2024-09-16T17:22:28.487 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:13b7075e:::benchmark_data_smithi039_455948_object292:head# 2024-09-16T17:22:28.487 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:14072aff:::benchmark_data_smithi039_455948_object266:head# 2024-09-16T17:22:28.488 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:14a5c78e:::benchmark_data_smithi039_455948_object46:head# 2024-09-16T17:22:28.489 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1561ff76:::benchmark_data_smithi039_455948_object155:head# 2024-09-16T17:22:28.490 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:15a4150c:::benchmark_data_smithi039_461800_object69:head# 2024-09-16T17:22:28.490 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:16d84de3:::benchmark_data_smithi039_455948_object200:head# 2024-09-16T17:22:28.491 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:19661353:::benchmark_data_smithi039_461800_object51:head# 2024-09-16T17:22:28.491 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1a3cf39c:::benchmark_data_smithi039_455948_object177:head# 2024-09-16T17:22:28.492 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1ae97a14:::benchmark_data_smithi039_455948_object69:head# 2024-09-16T17:22:28.492 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1b1fa822:::benchmark_data_smithi039_461800_object143:head# 2024-09-16T17:22:28.493 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1bcacdd1:::benchmark_data_smithi039_461800_object98:head# 2024-09-16T17:22:28.493 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1dda2954:::benchmark_data_smithi039_461800_object33:head# 2024-09-16T17:22:28.494 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1df9e5c8:::benchmark_data_smithi039_455948_object115:head# 2024-09-16T17:22:28.494 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1e34d247:::benchmark_data_smithi039_461800_object92:head# 2024-09-16T17:22:28.495 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1e6aee90:::benchmark_data_smithi039_455948_object312:head# 2024-09-16T17:22:28.495 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1e9dda24:::benchmark_data_smithi039_455948_object173:head# 2024-09-16T17:22:28.495 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1ea64c7e:::benchmark_data_smithi039_455948_object278:head# 2024-09-16T17:22:28.496 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1ebea92e:::benchmark_data_smithi039_461800_object175:head# 2024-09-16T17:22:28.497 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1fb0f211:::benchmark_data_smithi039_455948_object109:head# 2024-09-16T17:22:28.497 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1fbf7b6c:::benchmark_data_smithi039_455948_object207:head# 2024-09-16T17:22:28.498 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1fd4abd6:::benchmark_data_smithi039_455948_object142:head# 2024-09-16T17:22:28.498 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:208ac2b2:::benchmark_data_smithi039_461800_object130:head# 2024-09-16T17:22:28.498 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:20997167:::benchmark_data_smithi039_455948_object56:head# 2024-09-16T17:22:28.499 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:227f4aa5:::benchmark_data_smithi039_455948_object260:head# 2024-09-16T17:22:28.499 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:241b6c7b:::benchmark_data_smithi039_461800_object55:head# 2024-09-16T17:22:28.500 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:241f88ec:::benchmark_data_smithi039_455948_object118:head# 2024-09-16T17:22:28.500 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2464192d:::benchmark_data_smithi039_455948_object67:head# 2024-09-16T17:22:28.501 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:255d9356:::benchmark_data_smithi039_455948_object19:head# 2024-09-16T17:22:28.501 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:260b2bbc:::benchmark_data_smithi039_455948_object79:head# 2024-09-16T17:22:28.502 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:27914c55:::benchmark_data_smithi039_455948_object135:head# 2024-09-16T17:22:28.503 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:279e6bbf:::benchmark_data_smithi039_455948_object247:head# 2024-09-16T17:22:28.503 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:28d56bea:::benchmark_data_smithi039_455948_object11:head# 2024-09-16T17:22:28.503 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:294aa246:::benchmark_data_smithi039_455948_object38:head# 2024-09-16T17:22:28.503 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:29a8d043:::benchmark_data_smithi039_461800_object94:head# 2024-09-16T17:22:28.504 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:29d21410:::benchmark_data_smithi039_461800_object21:head# 2024-09-16T17:22:28.505 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:29d337cd:::benchmark_data_smithi039_461800_object24:head# 2024-09-16T17:22:28.505 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2a8d03cc:::benchmark_data_smithi039_455948_object197:head# 2024-09-16T17:22:28.506 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2ba0cc00:::benchmark_data_smithi039_461800_object39:head# 2024-09-16T17:22:28.506 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2c3b99e3:::benchmark_data_smithi039_461800_object108:head# 2024-09-16T17:22:28.507 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2cdc5a61:::benchmark_data_smithi039_455948_object87:head# 2024-09-16T17:22:28.507 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2de2bac8:::benchmark_data_smithi039_461800_object155:head# 2024-09-16T17:22:28.508 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2e4027e4:::benchmark_data_smithi039_461800_object70:head# 2024-09-16T17:22:28.508 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2e79016e:::benchmark_data_smithi039_461800_object8:head# 2024-09-16T17:22:28.508 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2ec5c54e:::benchmark_data_smithi039_461800_object74:head# 2024-09-16T17:22:28.509 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2f2af160:::benchmark_data_smithi039_455948_object63:head# 2024-09-16T17:22:28.509 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2f6ad66d:::benchmark_data_smithi039_455948_object237:head# 2024-09-16T17:22:28.510 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2f7d0174:::benchmark_data_smithi039_461800_object146:head# 2024-09-16T17:22:28.510 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2fa31139:::benchmark_data_smithi039_455948_object36:head# 2024-09-16T17:22:28.511 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2faa5036:::benchmark_data_smithi039_455948_object43:head# 2024-09-16T17:22:28.511 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3061c169:::benchmark_data_smithi039_455948_object271:head# 2024-09-16T17:22:28.512 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:30da3c6c:::benchmark_data_smithi039_455948_object193:head# 2024-09-16T17:22:28.512 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3105728e:::benchmark_data_smithi039_455948_object148:head# 2024-09-16T17:22:28.512 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3173dfe0:::benchmark_data_smithi039_455948_object188:head# 2024-09-16T17:22:28.513 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:334e8142:::benchmark_data_smithi039_461800_object78:head# 2024-09-16T17:22:28.514 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:335c1a68:::benchmark_data_smithi039_455948_object128:head# 2024-09-16T17:22:28.515 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:34153b00:::benchmark_data_smithi039_461800_object136:head# 2024-09-16T17:22:28.515 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:34d3b2fb:::benchmark_data_smithi039_461800_object142:head# 2024-09-16T17:22:28.515 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3509581c:::benchmark_data_smithi039_461800_object153:head# 2024-09-16T17:22:28.516 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:351c3af9:::benchmark_data_smithi039_455948_object72:head# 2024-09-16T17:22:28.516 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:369b8773:::benchmark_data_smithi039_461800_object133:head# 2024-09-16T17:22:28.517 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:36bb23c0:::benchmark_data_smithi039_455948_object101:head# 2024-09-16T17:22:28.517 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:375cb5d3:::benchmark_data_smithi039_455948_object275:head# 2024-09-16T17:22:28.518 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:37a5f014:::benchmark_data_smithi039_455948_object204:head# 2024-09-16T17:22:28.519 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:38196426:::benchmark_data_smithi039_461800_object30:head# 2024-09-16T17:22:28.519 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:39160a3b:::benchmark_data_smithi039_461800_object75:head# 2024-09-16T17:22:28.520 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3a6cafe9:::benchmark_data_smithi039_455948_object6:head# 2024-09-16T17:22:28.520 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3aab4211:::benchmark_data_smithi039_461800_object161:head# 2024-09-16T17:22:28.520 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3b836eaa:::benchmark_data_smithi039_461800_object154:head# 2024-09-16T17:22:28.521 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3c81aa17:::benchmark_data_smithi039_455948_object280:head# 2024-09-16T17:22:28.521 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3cd1f324:::benchmark_data_smithi039_455948_object202:head# 2024-09-16T17:22:28.522 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3d03a78e:::benchmark_data_smithi039_461800_object171:head# 2024-09-16T17:22:28.523 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3e761f82:::benchmark_data_smithi039_461800_object9:head# 2024-09-16T17:22:28.523 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3eaccc36:::benchmark_data_smithi039_455948_object144:head# 2024-09-16T17:22:28.523 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3f37ec22:::benchmark_data_smithi039_455948_object161:head# 2024-09-16T17:22:28.524 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3fc02a40:::benchmark_data_smithi039_455948_object74:head# 2024-09-16T17:22:28.524 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4003301f:::benchmark_data_smithi039_455948_object139:head# 2024-09-16T17:22:28.525 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4117f431:::benchmark_data_smithi039_455948_object234:head# 2024-09-16T17:22:28.525 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:41facc71:::benchmark_data_smithi039_455948_object198:head# 2024-09-16T17:22:28.525 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:430c10f6:::benchmark_data_smithi039_461800_object124:head# 2024-09-16T17:22:28.526 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4411f440:::benchmark_data_smithi039_455948_object297:head# 2024-09-16T17:22:28.527 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:44802c6d:::benchmark_data_smithi039_455948_object119:head# 2024-09-16T17:22:28.527 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:45c521fb:::benchmark_data_smithi039_455948_object249:head# 2024-09-16T17:22:28.528 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:45c56cfe:::benchmark_data_smithi039_461800_object80:head# 2024-09-16T17:22:28.528 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:469ace3a:::benchmark_data_smithi039_455948_object206:head# 2024-09-16T17:22:28.529 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:47970e62:::benchmark_data_smithi039_455948_object210:head# 2024-09-16T17:22:28.529 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:47b13862:::benchmark_data_smithi039_455948_object138:head# 2024-09-16T17:22:28.530 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4801d23d:::benchmark_data_smithi039_455948_object85:head# 2024-09-16T17:22:28.530 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4817f58f:::benchmark_data_smithi039_461800_object18:head# 2024-09-16T17:22:28.531 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4899b9aa:::benchmark_data_smithi039_461800_object79:head# 2024-09-16T17:22:28.531 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:491895da:::benchmark_data_smithi039_455948_object169:head# 2024-09-16T17:22:28.532 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:49285683:::benchmark_data_smithi039_461800_object47:head# 2024-09-16T17:22:28.532 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4935d551:::benchmark_data_smithi039_455948_object251:head# 2024-09-16T17:22:28.533 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:496b18ad:::benchmark_data_smithi039_455948_object253:head# 2024-09-16T17:22:28.533 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4a184ab0:::benchmark_data_smithi039_455948_object158:head# 2024-09-16T17:22:28.534 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4a1d7637:::benchmark_data_smithi039_455948_object183:head# 2024-09-16T17:22:28.534 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4ac873c7:::benchmark_data_smithi039_455948_object37:head# 2024-09-16T17:22:28.535 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4bc4a2e0:::benchmark_data_smithi039_455948_object49:head# 2024-09-16T17:22:28.535 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4c4cedac:::benchmark_data_smithi039_461800_object120:head# 2024-09-16T17:22:28.536 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4c531f09:::benchmark_data_smithi039_461800_object23:head# 2024-09-16T17:22:28.536 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4c5a7992:::benchmark_data_smithi039_455948_object221:head# 2024-09-16T17:22:28.537 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4c605a30:::benchmark_data_smithi039_461800_object100:head# 2024-09-16T17:22:28.537 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4d1c2c0b:::benchmark_data_smithi039_461800_object73:head# 2024-09-16T17:22:28.538 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4eaae14b:::benchmark_data_smithi039_461800_object67:head# 2024-09-16T17:22:28.539 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4f03ab2e:::benchmark_data_smithi039_455948_object318:head# 2024-09-16T17:22:28.539 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4f301666:::benchmark_data_smithi039_455948_object228:head# 2024-09-16T17:22:28.539 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4f3a0a7a:::benchmark_data_smithi039_455948_object240:head# 2024-09-16T17:22:28.540 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4fb8888a:::benchmark_data_smithi039_455948_object203:head# 2024-09-16T17:22:28.540 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:50a96376:::benchmark_data_smithi039_455948_object96:head# 2024-09-16T17:22:28.541 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:51ece5aa:::benchmark_data_smithi039_461800_object144:head# 2024-09-16T17:22:28.541 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5244e47b:::benchmark_data_smithi039_455948_object62:head# 2024-09-16T17:22:28.542 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:52d42a8e:::benchmark_data_smithi039_455948_object48:head# 2024-09-16T17:22:28.542 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:52e6e778:::benchmark_data_smithi039_461800_object105:head# 2024-09-16T17:22:28.543 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:52ede5be:::benchmark_data_smithi039_455948_object12:head# 2024-09-16T17:22:28.543 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:53a105b2:::benchmark_data_smithi039_455948_object311:head# 2024-09-16T17:22:28.544 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:53cb9fd2:::benchmark_data_smithi039_455948_object190:head# 2024-09-16T17:22:28.544 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:53ed147c:::benchmark_data_smithi039_455948_object31:head# 2024-09-16T17:22:28.545 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:54d49a7f:::benchmark_data_smithi039_455948_object28:head# 2024-09-16T17:22:28.546 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:54e04d99:::benchmark_data_smithi039_455948_object283:head# 2024-09-16T17:22:28.546 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:54e85f22:::benchmark_data_smithi039_455948_object23:head# 2024-09-16T17:22:28.546 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:55cbeda0:::benchmark_data_smithi039_461800_object123:head# 2024-09-16T17:22:28.547 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5651eea5:::benchmark_data_smithi039_455948_object18:head# 2024-09-16T17:22:28.547 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:57210b16:::benchmark_data_smithi039_461800_object72:head# 2024-09-16T17:22:28.548 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:57dabe70:::benchmark_data_smithi039_455948_object218:head# 2024-09-16T17:22:28.548 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5801fe81:::benchmark_data_smithi039_455948_object75:head# 2024-09-16T17:22:28.549 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:58d2c70a:::benchmark_data_smithi039_455948_object0:head# 2024-09-16T17:22:28.550 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:59614fc8:::benchmark_data_smithi039_455948_object76:head# 2024-09-16T17:22:28.550 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:598ce77d:::benchmark_data_smithi039_461800_object28:head# 2024-09-16T17:22:28.550 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:59c6e486:::benchmark_data_smithi039_461800_object164:head# 2024-09-16T17:22:28.551 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5a3d91e5:::benchmark_data_smithi039_455948_object108:head# 2024-09-16T17:22:28.551 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5a481a4a:::benchmark_data_smithi039_461800_object88:head# 2024-09-16T17:22:28.552 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5a7ae6c2:::benchmark_data_smithi039_455948_object100:head# 2024-09-16T17:22:28.552 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5c28febb:::benchmark_data_smithi039_455948_object156:head# 2024-09-16T17:22:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5d03c722:::benchmark_data_smithi039_461800_object141:head# 2024-09-16T17:22:28.553 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5db907e9:::benchmark_data_smithi039_455948_object121:head# 2024-09-16T17:22:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5ded13e4:::benchmark_data_smithi039_455948_object90:head# 2024-09-16T17:22:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5e428851:::benchmark_data_smithi039_455948_object309:head# 2024-09-16T17:22:28.554 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5e629290:::benchmark_data_smithi039_455948_object20:head# 2024-09-16T17:22:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5ee4ed35:::benchmark_data_smithi039_461800_object139:head# 2024-09-16T17:22:28.555 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5efeb4ae:::benchmark_data_smithi039_455948_object153:head# 2024-09-16T17:22:28.556 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5fd55c10:::benchmark_data_smithi039_461800_object97:head# 2024-09-16T17:22:28.557 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5ff59d48:::benchmark_data_smithi039_455948_object16:head# 2024-09-16T17:22:28.557 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:606af174:::benchmark_data_smithi039_461800_object148:head# 2024-09-16T17:22:28.558 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6114f880:::benchmark_data_smithi039_461800_object90:head# 2024-09-16T17:22:28.558 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6137bcba:::benchmark_data_smithi039_455948_object120:head# 2024-09-16T17:22:28.559 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6154b590:::benchmark_data_smithi039_455948_object58:head# 2024-09-16T17:22:28.559 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:61a5de33:::benchmark_data_smithi039_461800_object149:head# 2024-09-16T17:22:28.560 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6280b741:::benchmark_data_smithi039_455948_object250:head# 2024-09-16T17:22:28.560 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:62961663:::benchmark_data_smithi039_455948_object296:head# 2024-09-16T17:22:28.560 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:633b496b:::benchmark_data_smithi039_461800_object126:head# 2024-09-16T17:22:28.561 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:635c8e6d:::benchmark_data_smithi039_461800_object56:head# 2024-09-16T17:22:28.561 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:642ab5c2:::benchmark_data_smithi039_455948_object41:head# 2024-09-16T17:22:28.562 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:64aa8ba8:::benchmark_data_smithi039_455948_object71:head# 2024-09-16T17:22:28.562 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:654ddc28:::benchmark_data_smithi039_455948_object147:head# 2024-09-16T17:22:28.562 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:656a01f0:::benchmark_data_smithi039_455948_object316:head# 2024-09-16T17:22:28.563 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6667fb4d:::benchmark_data_smithi039_455948_object252:head# 2024-09-16T17:22:28.563 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:66719837:::benchmark_data_smithi039_455948_object179:head# 2024-09-16T17:22:28.564 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:66a455f7:::benchmark_data_smithi039_461800_object31:head# 2024-09-16T17:22:28.565 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:67fc60d1:::benchmark_data_smithi039_455948_object103:head# 2024-09-16T17:22:28.565 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:67fc91c1:::benchmark_data_smithi039_455948_object52:head# 2024-09-16T17:22:28.566 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6815ad2e:::benchmark_data_smithi039_455948_object196:head# 2024-09-16T17:22:28.566 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:69293e2b:::benchmark_data_smithi039_455948_object290:head# 2024-09-16T17:22:28.567 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6ae4f661:::benchmark_data_smithi039_455948_object241:head# 2024-09-16T17:22:28.567 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6cb11b42:::benchmark_data_smithi039_455948_object285:head# 2024-09-16T17:22:28.567 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6ce0dbf0:::benchmark_data_smithi039_455948_object268:head# 2024-09-16T17:22:28.568 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6d2fe8c5:::benchmark_data_smithi039_455948_object273:head# 2024-09-16T17:22:28.568 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6d540a4e:::benchmark_data_smithi039_455948_object8:head# 2024-09-16T17:22:28.568 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6d724163:::benchmark_data_smithi039_455948_object269:head# 2024-09-16T17:22:28.569 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6e38bb84:::benchmark_data_smithi039_461800_object102:head# 2024-09-16T17:22:28.570 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6e61b9d5:::benchmark_data_smithi039_455948_object68:head# 2024-09-16T17:22:28.570 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6f095114:::benchmark_data_smithi039_455948_object94:head# 2024-09-16T17:22:28.571 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:70691fc8:::benchmark_data_smithi039_455948_object145:head# 2024-09-16T17:22:28.571 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:708034b3:::benchmark_data_smithi039_455948_object276:head# 2024-09-16T17:22:28.572 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:70f55f31:::benchmark_data_smithi039_455948_object239:head# 2024-09-16T17:22:28.572 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:710d1bd1:::benchmark_data_smithi039_461800_object63:head# 2024-09-16T17:22:28.573 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:713ede28:::benchmark_data_smithi039_461800_object99:head# 2024-09-16T17:22:28.574 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7149ae21:::benchmark_data_smithi039_461800_object84:head# 2024-09-16T17:22:28.574 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:718c00a4:::benchmark_data_smithi039_461800_object127:head# 2024-09-16T17:22:28.575 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:71b1cb82:::benchmark_data_smithi039_455948_object65:head# 2024-09-16T17:22:28.575 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:71ce8906:::benchmark_data_smithi039_455948_object78:head# 2024-09-16T17:22:28.576 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:72804f95:::benchmark_data_smithi039_461800_object22:head# 2024-09-16T17:22:28.576 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:72fdddd5:::benchmark_data_smithi039_455948_object86:head# 2024-09-16T17:22:28.577 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:732a665c:::benchmark_data_smithi039_461800_object26:head# 2024-09-16T17:22:28.577 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7377856f:::benchmark_data_smithi039_461800_object15:head# 2024-09-16T17:22:28.577 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:73b633f2:::benchmark_data_smithi039_455948_object150:head# 2024-09-16T17:22:28.578 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:754b4658:::benchmark_data_smithi039_455948_object77:head# 2024-09-16T17:22:28.578 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:75a63f9f:::benchmark_data_smithi039_455948_object226:head# 2024-09-16T17:22:28.579 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:75f9bf67:::benchmark_data_smithi039_455948_object242:head# 2024-09-16T17:22:28.579 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:76321804:::benchmark_data_smithi039_461800_object49:head# 2024-09-16T17:22:28.580 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:764ab525:::benchmark_data_smithi039_455948_object317:head# 2024-09-16T17:22:28.580 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:768e2761:::benchmark_data_smithi039_455948_object238:head# 2024-09-16T17:22:28.581 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:77182e0f:::benchmark_data_smithi039_455948_object140:head# 2024-09-16T17:22:28.581 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7739f249:::benchmark_data_smithi039_455948_object34:head# 2024-09-16T17:22:28.581 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:784592ec:::benchmark_data_smithi039_455948_object33:head# 2024-09-16T17:22:28.581 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:78c85cc5:::benchmark_data_smithi039_461800_object147:head# 2024-09-16T17:22:28.582 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7990ab76:::benchmark_data_smithi039_461800_object167:head# 2024-09-16T17:22:28.583 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7995bd94:::benchmark_data_smithi039_461800_object57:head# 2024-09-16T17:22:28.583 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7a597a33:::benchmark_data_smithi039_455948_object274:head# 2024-09-16T17:22:28.584 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7b291d50:::benchmark_data_smithi039_455948_object220:head# 2024-09-16T17:22:28.584 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7b313b26:::benchmark_data_smithi039_461800_object82:head# 2024-09-16T17:22:28.585 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7bf1daea:::benchmark_data_smithi039_461800_object162:head# 2024-09-16T17:22:28.585 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7c07aaad:::benchmark_data_smithi039_455948_object91:head# 2024-09-16T17:22:28.585 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7cd2d272:::benchmark_data_smithi039_455948_object53:head# 2024-09-16T17:22:28.586 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7d0d8840:::benchmark_data_smithi039_455948_object264:head# 2024-09-16T17:22:28.586 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7d2f1a11:::benchmark_data_smithi039_461800_object163:head# 2024-09-16T17:22:28.586 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7d342ae1:::benchmark_data_smithi039_455948_object47:head# 2024-09-16T17:22:28.587 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7ee172f3:::benchmark_data_smithi039_455948_object304:head# 2024-09-16T17:22:28.587 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7fe7c24d:::benchmark_data_smithi039_455948_object88:head# 2024-09-16T17:22:28.588 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:808f8c5a:::benchmark_data_smithi039_461800_object4:head# 2024-09-16T17:22:28.588 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:80b28505:::benchmark_data_smithi039_461800_object59:head# 2024-09-16T17:22:28.589 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:81444e03:::benchmark_data_smithi039_455948_object191:head# 2024-09-16T17:22:28.605 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:81524550:::benchmark_data_smithi039_455948_object291:head# 2024-09-16T17:22:28.606 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:81688782:::benchmark_data_smithi039_455948_object114:head# 2024-09-16T17:22:28.608 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:81bb4070:::benchmark_data_smithi039_461800_object46:head# 2024-09-16T17:22:28.608 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:82380bb6:::benchmark_data_smithi039_461800_object12:head# 2024-09-16T17:22:28.609 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8340282c:::benchmark_data_smithi039_455948_object261:head# 2024-09-16T17:22:28.609 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:84461d11:::benchmark_data_smithi039_455948_object236:head# 2024-09-16T17:22:28.610 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:85152b24:::benchmark_data_smithi039_461800_object125:head# 2024-09-16T17:22:28.610 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:85bbfc69:::benchmark_data_smithi039_455948_object30:head# 2024-09-16T17:22:28.611 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:861aea5f:::benchmark_data_smithi039_455948_object258:head# 2024-09-16T17:22:28.611 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8663e6e8:::benchmark_data_smithi039_461800_object112:head# 2024-09-16T17:22:28.612 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:869e8a5e:::benchmark_data_smithi039_455948_object167:head# 2024-09-16T17:22:28.613 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:872ef1cc:::benchmark_data_smithi039_455948_object195:head# 2024-09-16T17:22:28.613 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:87707c04:::benchmark_data_smithi039_461800_object38:head# 2024-09-16T17:22:28.613 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8923ed3b:::benchmark_data_smithi039_455948_object5:head# 2024-09-16T17:22:28.614 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:899902b6:::benchmark_data_smithi039_461800_object13:head# 2024-09-16T17:22:28.614 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:89be3f00:::benchmark_data_smithi039_461800_object2:head# 2024-09-16T17:22:28.615 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8a3ec687:::benchmark_data_smithi039_455948_object214:head# 2024-09-16T17:22:28.615 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8aec744d:::benchmark_data_smithi039_455948_object254:head# 2024-09-16T17:22:28.615 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8c1746e2:::benchmark_data_smithi039_461800_object19:head# 2024-09-16T17:22:28.616 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8c182888:::benchmark_data_smithi039_461800_object60:head# 2024-09-16T17:22:28.616 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8c74c0d6:::benchmark_data_smithi039_455948_object116:head# 2024-09-16T17:22:28.617 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8c7d0d6d:::benchmark_data_smithi039_461800_object35:head# 2024-09-16T17:22:28.617 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8eb4657a:::benchmark_data_smithi039_455948_object163:head# 2024-09-16T17:22:28.617 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8ebb2bdf:::benchmark_data_smithi039_461800_object132:head# 2024-09-16T17:22:28.618 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92b8125e:::benchmark_data_smithi039_455948_object165:head# 2024-09-16T17:22:28.618 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92dd79ef:::benchmark_data_smithi039_455948_object130:head# 2024-09-16T17:22:28.619 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92ec70a0:::benchmark_data_smithi039_455948_object170:head# 2024-09-16T17:22:28.619 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92f0e148:::benchmark_data_smithi039_455948_object223:head# 2024-09-16T17:22:28.620 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:930eb7ac:::benchmark_data_smithi039_455948_object262:head# 2024-09-16T17:22:28.620 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:943cbef4:::benchmark_data_smithi039_461800_object66:head# 2024-09-16T17:22:28.621 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:972245e1:::benchmark_data_smithi039_455948_object7:head# 2024-09-16T17:22:28.622 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:972cbf49:::benchmark_data_smithi039_455948_object3:head# 2024-09-16T17:22:28.622 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:97809665:::benchmark_data_smithi039_461800_object176:head# 2024-09-16T17:22:28.623 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:984c6070:::benchmark_data_smithi039_455948_object1:head# 2024-09-16T17:22:28.623 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:98946638:::benchmark_data_smithi039_455948_object219:head# 2024-09-16T17:22:28.624 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:98a59f74:::benchmark_data_smithi039_455948_object216:head# 2024-09-16T17:22:28.625 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:98b0eb5d:::benchmark_data_smithi039_455948_object131:head# 2024-09-16T17:22:28.625 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:99654740:::benchmark_data_smithi039_455948_object141:head# 2024-09-16T17:22:28.625 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9aaac537:::benchmark_data_smithi039_461800_object58:head# 2024-09-16T17:22:28.626 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9b110e6f:::benchmark_data_smithi039_455948_object55:head# 2024-09-16T17:22:28.626 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9c86eb42:::benchmark_data_smithi039_455948_object287:head# 2024-09-16T17:22:28.627 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9c901805:::benchmark_data_smithi039_455948_object59:head# 2024-09-16T17:22:28.627 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9cc334c3:::benchmark_data_smithi039_461800_object87:head# 2024-09-16T17:22:28.628 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9d050ed9:::benchmark_data_smithi039_455948_object284:head# 2024-09-16T17:22:28.628 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9e116042:::benchmark_data_smithi039_455948_object286:head# 2024-09-16T17:22:28.628 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9eac4879:::benchmark_data_smithi039_455948_object289:head# 2024-09-16T17:22:28.629 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9f0bc29c:::benchmark_data_smithi039_455948_object257:head# 2024-09-16T17:22:28.630 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9f254ec7:::benchmark_data_smithi039_461800_object1:head# 2024-09-16T17:22:28.630 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9fe1d67f:::benchmark_data_smithi039_461800_object0:head# 2024-09-16T17:22:28.631 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a0acf962:::benchmark_data_smithi039_455948_object61:head# 2024-09-16T17:22:28.632 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a0b1ff33:::benchmark_data_smithi039_455948_object133:head# 2024-09-16T17:22:28.632 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a229396d:::benchmark_data_smithi039_461800_object135:head# 2024-09-16T17:22:28.633 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a299f295:::benchmark_data_smithi039_455948_object176:head# 2024-09-16T17:22:28.633 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a3610da1:::benchmark_data_smithi039_455948_object149:head# 2024-09-16T17:22:28.634 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a3616c00:::benchmark_data_smithi039_455948_object246:head# 2024-09-16T17:22:28.634 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a37a51d4:::benchmark_data_smithi039_461800_object61:head# 2024-09-16T17:22:28.635 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a3d8a56b:::benchmark_data_smithi039_455948_object73:head# 2024-09-16T17:22:28.636 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a437720d:::benchmark_data_smithi039_455948_object13:head# 2024-09-16T17:22:28.636 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a44c4b31:::benchmark_data_smithi039_461800_object14:head# 2024-09-16T17:22:28.637 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a5276dd6:::benchmark_data_smithi039_455948_object2:head# 2024-09-16T17:22:28.637 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a5e30e12:::benchmark_data_smithi039_455948_object17:head# 2024-09-16T17:22:28.637 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a5e964e3:::benchmark_data_smithi039_455948_object45:head# 2024-09-16T17:22:28.638 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a5f6a828:::benchmark_data_smithi039_455948_object25:head# 2024-09-16T17:22:28.638 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a63079ff:::benchmark_data_smithi039_455948_object265:head# 2024-09-16T17:22:28.639 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a64c1b86:::benchmark_data_smithi039_455948_object288:head# 2024-09-16T17:22:28.639 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a668a0c5:::benchmark_data_smithi039_461800_object20:head# 2024-09-16T17:22:28.640 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a6698c47:::benchmark_data_smithi039_455948_object259:head# 2024-09-16T17:22:28.640 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a69d0e24:::benchmark_data_smithi039_455948_object113:head# 2024-09-16T17:22:28.641 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a6c2f07b:::benchmark_data_smithi039_455948_object212:head# 2024-09-16T17:22:28.641 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a8211e63:::benchmark_data_smithi039_455948_object281:head# 2024-09-16T17:22:28.642 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a9b29118:::benchmark_data_smithi039_461800_object52:head# 2024-09-16T17:22:28.642 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a9f1c778:::benchmark_data_smithi039_461800_object107:head# 2024-09-16T17:22:28.643 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:aa4113d5:::benchmark_data_smithi039_461800_object104:head# 2024-09-16T17:22:28.643 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:aba735d4:::benchmark_data_smithi039_461800_object27:head# 2024-09-16T17:22:28.644 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:abd255dc:::benchmark_data_smithi039_461800_object83:head# 2024-09-16T17:22:28.644 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ad13fc26:::benchmark_data_smithi039_455948_object110:head# 2024-09-16T17:22:28.645 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ad8239d1:::benchmark_data_smithi039_455948_object232:head# 2024-09-16T17:22:28.645 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ad91945e:::benchmark_data_smithi039_455948_object93:head# 2024-09-16T17:22:28.646 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:adaffd31:::benchmark_data_smithi039_455948_object26:head# 2024-09-16T17:22:28.646 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ae18ef59:::benchmark_data_smithi039_461800_object17:head# 2024-09-16T17:22:28.646 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:aeb14177:::benchmark_data_smithi039_455948_object29:head# 2024-09-16T17:22:28.647 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:aecd8d89:::benchmark_data_smithi039_455948_object174:head# 2024-09-16T17:22:28.647 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b03ef840:::benchmark_data_smithi039_455948_object295:head# 2024-09-16T17:22:28.648 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b089b62d:::benchmark_data_smithi039_461800_object7:head# 2024-09-16T17:22:28.648 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b0c8cac8:::benchmark_data_smithi039_455948_object117:head# 2024-09-16T17:22:28.649 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b134289d:::benchmark_data_smithi039_461800_object150:head# 2024-09-16T17:22:28.649 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b137eb01:::benchmark_data_smithi039_455948_object211:head# 2024-09-16T17:22:28.650 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b2347b11:::benchmark_data_smithi039_455948_object301:head# 2024-09-16T17:22:28.650 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b249856d:::benchmark_data_smithi039_455948_object235:head# 2024-09-16T17:22:28.651 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b2bcdbb8:::benchmark_data_smithi039_461800_object114:head# 2024-09-16T17:22:28.651 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b3253c9c:::benchmark_data_smithi039_455948_object255:head# 2024-09-16T17:22:28.652 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b3578fa0:::benchmark_data_smithi039_461800_object121:head# 2024-09-16T17:22:28.652 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b4899b76:::benchmark_data_smithi039_461800_object165:head# 2024-09-16T17:22:28.653 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b522d3d2:::benchmark_data_smithi039_461800_object152:head# 2024-09-16T17:22:28.653 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b5b57515:::benchmark_data_smithi039_455948_object129:head# 2024-09-16T17:22:28.654 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b5ebc165:::benchmark_data_smithi039_461800_object64:head# 2024-09-16T17:22:28.654 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b6a40ada:::benchmark_data_smithi039_461800_object76:head# 2024-09-16T17:22:28.655 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b6c46095:::benchmark_data_smithi039_455948_object178:head# 2024-09-16T17:22:28.655 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b6ce9049:::benchmark_data_smithi039_455948_object172:head# 2024-09-16T17:22:28.655 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b6cf5c8c:::benchmark_data_smithi039_455948_object231:head# 2024-09-16T17:22:28.656 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b7c8f422:::benchmark_data_smithi039_455948_object112:head# 2024-09-16T17:22:28.656 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b811e841:::benchmark_data_smithi039_461800_object134:head# 2024-09-16T17:22:28.657 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-09-16T17:22:28.657 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b8b98acb:::benchmark_data_smithi039_455948_object313:head# 2024-09-16T17:22:28.658 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b9991d10:::benchmark_data_smithi039_455948_object154:head# 2024-09-16T17:22:28.658 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b9d0eac5:::benchmark_data_smithi039_461800_object145:head# 2024-09-16T17:22:28.658 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b9f36334:::benchmark_data_smithi039_461800_object91:head# 2024-09-16T17:22:28.659 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bad4f01e:::benchmark_data_smithi039_461800_object29:head# 2024-09-16T17:22:28.660 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:baf26fd7:::benchmark_data_smithi039_455948_object134:head# 2024-09-16T17:22:28.660 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bb71b590:::benchmark_data_smithi039_455948_object314:head# 2024-09-16T17:22:28.660 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bba0e1da:::benchmark_data_smithi039_461800_object6:head# 2024-09-16T17:22:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bba586dd:::benchmark_data_smithi039_461800_object81:head# 2024-09-16T17:22:28.661 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bc052407:::benchmark_data_smithi039_455948_object270:head# 2024-09-16T17:22:28.662 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bcff6325:::benchmark_data_smithi039_455948_object315:head# 2024-09-16T17:22:28.662 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bd19ab08:::benchmark_data_smithi039_455948_object81:head# 2024-09-16T17:22:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:be7390f5:::benchmark_data_smithi039_455948_object83:head# 2024-09-16T17:22:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bf8a7830:::benchmark_data_smithi039_461800_object41:head# 2024-09-16T17:22:28.663 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c07324f3:::benchmark_data_smithi039_455948_object302:head# 2024-09-16T17:22:28.664 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c094cc24:::benchmark_data_smithi039_461800_object89:head# 2024-09-16T17:22:28.664 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c0d972fb:::benchmark_data_smithi039_455948_object209:head# 2024-09-16T17:22:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c1b3a68a:::benchmark_data_smithi039_461800_object160:head# 2024-09-16T17:22:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c34a47c1:::benchmark_data_smithi039_455948_object319:head# 2024-09-16T17:22:28.665 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c38689dc:::benchmark_data_smithi039_455948_object50:head# 2024-09-16T17:22:28.666 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c3893c52:::benchmark_data_smithi039_455948_object66:head# 2024-09-16T17:22:28.666 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c464c3ce:::benchmark_data_smithi039_455948_object106:head# 2024-09-16T17:22:28.667 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c496e5b5:::benchmark_data_smithi039_461800_object110:head# 2024-09-16T17:22:28.667 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c4a7d2e8:::benchmark_data_smithi039_461800_object96:head# 2024-09-16T17:22:28.668 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c56ee08e:::benchmark_data_smithi039_455948_object125:head# 2024-09-16T17:22:28.668 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c6555459:::benchmark_data_smithi039_455948_object282:head# 2024-09-16T17:22:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c6ec286a:::benchmark_data_smithi039_461800_object174:head# 2024-09-16T17:22:28.669 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c70cd8e6:::benchmark_data_smithi039_455948_object181:head# 2024-09-16T17:22:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c72b9798:::benchmark_data_smithi039_455948_object186:head# 2024-09-16T17:22:28.670 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c7547bd4:::benchmark_data_smithi039_455948_object160:head# 2024-09-16T17:22:28.671 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c8cc5007:::benchmark_data_smithi039_461800_object128:head# 2024-09-16T17:22:28.671 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c8d50a66:::benchmark_data_smithi039_455948_object15:head# 2024-09-16T17:22:28.672 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c916bf36:::benchmark_data_smithi039_455948_object300:head# 2024-09-16T17:22:28.672 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c98347ce:::benchmark_data_smithi039_455948_object44:head# 2024-09-16T17:22:28.673 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ca283ebf:::benchmark_data_smithi039_455948_object245:head# 2024-09-16T17:22:28.673 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cb190b2d:::benchmark_data_smithi039_455948_object299:head# 2024-09-16T17:22:28.674 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cb44df0b:::benchmark_data_smithi039_455948_object102:head# 2024-09-16T17:22:28.674 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cb8dc5ee:::benchmark_data_smithi039_455948_object201:head# 2024-09-16T17:22:28.675 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cd24ea32:::benchmark_data_smithi039_455948_object42:head# 2024-09-16T17:22:28.675 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cd94f270:::benchmark_data_smithi039_455948_object124:head# 2024-09-16T17:22:28.675 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cdc23a9a:::benchmark_data_smithi039_461800_object32:head# 2024-09-16T17:22:28.676 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ce974f23:::benchmark_data_smithi039_455948_object159:head# 2024-09-16T17:22:28.677 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cebe2675:::benchmark_data_smithi039_461800_object140:head# 2024-09-16T17:22:28.677 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cf1dabff:::benchmark_data_smithi039_455948_object151:head# 2024-09-16T17:22:28.678 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cf56f22e:::benchmark_data_smithi039_455948_object294:head# 2024-09-16T17:22:28.679 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cfee26d6:::benchmark_data_smithi039_461800_object95:head# 2024-09-16T17:22:28.679 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d04eceaf:::benchmark_data_smithi039_455948_object39:head# 2024-09-16T17:22:28.680 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d06d3616:::benchmark_data_smithi039_455948_object24:head# 2024-09-16T17:22:28.680 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d08457ea:::benchmark_data_smithi039_455948_object4:head# 2024-09-16T17:22:28.681 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d0851a13:::benchmark_data_smithi039_455948_object306:head# 2024-09-16T17:22:28.681 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d08dfc7d:::benchmark_data_smithi039_455948_object89:head# 2024-09-16T17:22:28.682 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d0bdfad3:::benchmark_data_smithi039_461800_object109:head# 2024-09-16T17:22:28.682 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d1c0a25d:::benchmark_data_smithi039_455948_object105:head# 2024-09-16T17:22:28.683 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d2347595:::benchmark_data_smithi039_455948_object123:head# 2024-09-16T17:22:28.683 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d26aabdc:::benchmark_data_smithi039_461800_object168:head# 2024-09-16T17:22:28.684 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d2987685:::benchmark_data_smithi039_461800_object172:head# 2024-09-16T17:22:28.684 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d2a4d7e3:::benchmark_data_smithi039_461800_object106:head# 2024-09-16T17:22:28.685 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d2f20841:::benchmark_data_smithi039_455948_object95:head# 2024-09-16T17:22:28.685 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d33b721d:::benchmark_data_smithi039_455948_object244:head# 2024-09-16T17:22:28.686 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d37b01a7:::benchmark_data_smithi039_455948_object243:head# 2024-09-16T17:22:28.687 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d432d0ae:::benchmark_data_smithi039_455948_object99:head# 2024-09-16T17:22:28.687 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d4cfa604:::benchmark_data_smithi039_455948_object14:head# 2024-09-16T17:22:28.687 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d55839d8:::benchmark_data_smithi039_461800_object129:head# 2024-09-16T17:22:28.688 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d66a0831:::benchmark_data_smithi039_455948_object143:head# 2024-09-16T17:22:28.689 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d7aed95b:::benchmark_data_smithi039_461800_object71:head# 2024-09-16T17:22:28.689 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d7dff5fe:::benchmark_data_smithi039_455948_object51:head# 2024-09-16T17:22:28.690 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d81d2707:::benchmark_data_smithi039_455948_object248:head# 2024-09-16T17:22:28.690 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d8cdeb85:::benchmark_data_smithi039_455948_object279:head# 2024-09-16T17:22:28.691 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d95fa43b:::benchmark_data_smithi039_455948_object97:head# 2024-09-16T17:22:28.691 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:da4991aa:::benchmark_data_smithi039_461800_object54:head# 2024-09-16T17:22:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:db0a03c3:::benchmark_data_smithi039_455948_object35:head# 2024-09-16T17:22:28.692 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:db111d65:::benchmark_data_smithi039_461800_object86:head# 2024-09-16T17:22:28.693 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:db62d852:::benchmark_data_smithi039_461800_object159:head# 2024-09-16T17:22:28.693 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dbe2ac53:::benchmark_data_smithi039_455948_object92:head# 2024-09-16T17:22:28.694 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dc7e8981:::benchmark_data_smithi039_461800_object53:head# 2024-09-16T17:22:28.694 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dcfb2107:::benchmark_data_smithi039_461800_object118:head# 2024-09-16T17:22:28.695 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:de97e1fb:::benchmark_data_smithi039_461800_object3:head# 2024-09-16T17:22:28.695 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e003ea5b:::benchmark_data_smithi039_461800_object11:head# 2024-09-16T17:22:28.695 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e038408c:::benchmark_data_smithi039_461800_object40:head# 2024-09-16T17:22:28.696 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e05e2d0e:::benchmark_data_smithi039_455948_object127:head# 2024-09-16T17:22:28.696 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e0c79d68:::benchmark_data_smithi039_461800_object93:head# 2024-09-16T17:22:28.697 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e0f1cb6f:::benchmark_data_smithi039_461800_object169:head# 2024-09-16T17:22:28.697 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e18f9e35:::benchmark_data_smithi039_455948_object82:head# 2024-09-16T17:22:28.698 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e1a4e1b7:::benchmark_data_smithi039_455948_object189:head# 2024-09-16T17:22:28.698 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e2081651:::benchmark_data_smithi039_455948_object303:head# 2024-09-16T17:22:28.699 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e22d9d39:::benchmark_data_smithi039_461800_object44:head# 2024-09-16T17:22:28.699 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e3634a32:::benchmark_data_smithi039_461800_object48:head# 2024-09-16T17:22:28.699 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e36479d3:::benchmark_data_smithi039_455948_object277:head# 2024-09-16T17:22:28.699 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e4509956:::benchmark_data_smithi039_455948_object222:head# 2024-09-16T17:22:28.700 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e45ad9a8:::benchmark_data_smithi039_455948_object310:head# 2024-09-16T17:22:28.700 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e49f1f25:::benchmark_data_smithi039_455948_object60:head# 2024-09-16T17:22:28.701 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e65ee987:::benchmark_data_smithi039_461800_object5:head# 2024-09-16T17:22:28.702 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e7a4dce5:::benchmark_data_smithi039_461800_object34:head# 2024-09-16T17:22:28.702 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e8279365:::benchmark_data_smithi039_455948_object162:head# 2024-09-16T17:22:28.702 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e84b1af3:::benchmark_data_smithi039_455948_object168:head# 2024-09-16T17:22:28.703 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e860450e:::benchmark_data_smithi039_455948_object40:head# 2024-09-16T17:22:28.703 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e9546805:::benchmark_data_smithi039_455948_object84:head# 2024-09-16T17:22:28.703 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e9c31f48:::benchmark_data_smithi039_455948_object57:head# 2024-09-16T17:22:28.704 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e9db3ae8:::benchmark_data_smithi039_455948_object185:head# 2024-09-16T17:22:28.705 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ea00b668:::benchmark_data_smithi039_455948_object126:head# 2024-09-16T17:22:28.705 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ea038be8:::benchmark_data_smithi039_461800_object103:head# 2024-09-16T17:22:28.706 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ea427d4a:::benchmark_data_smithi039_461800_object25:head# 2024-09-16T17:22:28.706 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:eac552d0:::benchmark_data_smithi039_455948_object217:head# 2024-09-16T17:22:28.706 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ed73d1bd:::benchmark_data_smithi039_461800_object117:head# 2024-09-16T17:22:28.707 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ed7b1e48:::benchmark_data_smithi039_461800_object157:head# 2024-09-16T17:22:28.707 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ed7c17e0:::benchmark_data_smithi039_455948_object152:head# 2024-09-16T17:22:28.708 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:edc14e53:::benchmark_data_smithi039_455948_object230:head# 2024-09-16T17:22:28.709 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ee810f8e:::benchmark_data_smithi039_461800_object173:head# 2024-09-16T17:22:28.709 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ee9608a4:::benchmark_data_smithi039_461800_object65:head# 2024-09-16T17:22:28.710 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:eebb0eed:::benchmark_data_smithi039_461800_object137:head# 2024-09-16T17:22:28.710 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:efd04b1c:::benchmark_data_smithi039_455948_object225:head# 2024-09-16T17:22:28.711 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:efea2d35:::benchmark_data_smithi039_455948_object146:head# 2024-09-16T17:22:28.711 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:eff1383c:::benchmark_data_smithi039_461800_object166:head# 2024-09-16T17:22:28.712 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f01ad973:::benchmark_data_smithi039_461800_object131:head# 2024-09-16T17:22:28.712 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f1bb07dd:::benchmark_data_smithi039_455948_object298:head# 2024-09-16T17:22:28.712 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f1e99f50:::benchmark_data_smithi039_455948_object215:head# 2024-09-16T17:22:28.713 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f1f5e565:::benchmark_data_smithi039_455948_object307:head# 2024-09-16T17:22:28.713 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f20c71ff:::benchmark_data_smithi039_455948_object267:head# 2024-09-16T17:22:28.714 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f274efb2:::benchmark_data_smithi039_455948_object27:head# 2024-09-16T17:22:28.714 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f2a8576b:::benchmark_data_smithi039_455948_object54:head# 2024-09-16T17:22:28.714 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f2f0a09c:::benchmark_data_smithi039_455948_object227:head# 2024-09-16T17:22:28.715 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f39faa1c:::benchmark_data_smithi039_461800_object151:head# 2024-09-16T17:22:28.716 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f401f323:::benchmark_data_smithi039_461800_object36:head# 2024-09-16T17:22:28.717 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f5996306:::benchmark_data_smithi039_455948_object22:head# 2024-09-16T17:22:28.717 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f5dc5fcb:::benchmark_data_smithi039_461800_object156:head# 2024-09-16T17:22:28.718 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f5ffe507:::benchmark_data_smithi039_461800_object116:head# 2024-09-16T17:22:28.718 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f61ef65d:::benchmark_data_smithi039_455948_object111:head# 2024-09-16T17:22:28.719 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f621bc7f:::benchmark_data_smithi039_455948_object182:head# 2024-09-16T17:22:28.719 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f6edecad:::benchmark_data_smithi039_461800_object111:head# 2024-09-16T17:22:28.720 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f7044b00:::benchmark_data_smithi039_461800_object138:head# 2024-09-16T17:22:28.720 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f72c3135:::benchmark_data_smithi039_455948_object21:head# 2024-09-16T17:22:28.721 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f7b30eee:::benchmark_data_smithi039_455948_object194:head# 2024-09-16T17:22:28.721 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f97e448b:::benchmark_data_smithi039_455948_object10:head# 2024-09-16T17:22:28.722 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f98eff84:::benchmark_data_smithi039_461800_object16:head# 2024-09-16T17:22:28.722 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f9d8cd97:::benchmark_data_smithi039_455948_object199:head# 2024-09-16T17:22:28.723 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f9e3593c:::benchmark_data_smithi039_455948_object256:head# 2024-09-16T17:22:28.723 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fa9bf67e:::benchmark_data_smithi039_455948_object32:head# 2024-09-16T17:22:28.724 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fadac265:::benchmark_data_smithi039_455948_object305:head# 2024-09-16T17:22:28.724 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fadc5b0b:::benchmark_data_smithi039_455948_object104:head# 2024-09-16T17:22:28.725 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fc574797:::benchmark_data_smithi039_461800_object62:head# 2024-09-16T17:22:28.725 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fc984da2:::benchmark_data_smithi039_461800_object37:head# 2024-09-16T17:22:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fd05df8b:::benchmark_data_smithi039_455948_object70:head# 2024-09-16T17:22:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fdea465d:::benchmark_data_smithi039_455948_object107:head# 2024-09-16T17:22:28.727 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fe001570:::benchmark_data_smithi039_455948_object171:head# 2024-09-16T17:22:28.727 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ff9caaec:::benchmark_data_smithi039_455948_object263:head# 2024-09-16T17:22:28.728 INFO:tasks.workunit.client.0.smithi039.stderr:Export successful 2024-09-16T17:22:28.728 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-16T17:22:28.728 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-16T17:22:28.728 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-16T17:22:29.213 INFO:tasks.workunit.client.0.smithi039.stderr:Remove successful 2024-09-16T17:22:29.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:143: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-09-16T17:22:30.122 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-16T17:22:30.122 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.1 2024-09-16T17:22:30.122 INFO:tasks.workunit.client.0.smithi039.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-16T17:22:30.122 INFO:tasks.workunit.client.0.smithi039.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-09-16T17:22:30.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:144: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-16T17:22:31.125 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-16T17:22:31.125 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.0 2024-09-16T17:22:31.125 INFO:tasks.workunit.client.0.smithi039.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-16T17:22:31.126 INFO:tasks.workunit.client.0.smithi039.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-09-16T17:22:31.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:147: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-09-16T17:22:32.141 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-16T17:22:32.141 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.1 2024-09-16T17:22:32.142 INFO:tasks.workunit.client.0.smithi039.stdout:pool 1 pg_num 2 -> 1 2024-09-16T17:22:32.142 INFO:tasks.workunit.client.0.smithi039.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-16T17:22:32.142 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81444e03:::benchmark_data_smithi039_455948_object191:head# 2024-09-16T17:22:32.142 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.164 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81524550:::benchmark_data_smithi039_455948_object291:head# 2024-09-16T17:22:32.164 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.172 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81688782:::benchmark_data_smithi039_455948_object114:head# 2024-09-16T17:22:32.172 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.180 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8340282c:::benchmark_data_smithi039_455948_object261:head# 2024-09-16T17:22:32.180 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.189 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:84461d11:::benchmark_data_smithi039_455948_object236:head# 2024-09-16T17:22:32.189 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.197 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:85bbfc69:::benchmark_data_smithi039_455948_object30:head# 2024-09-16T17:22:32.197 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.205 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:861aea5f:::benchmark_data_smithi039_455948_object258:head# 2024-09-16T17:22:32.205 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.214 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:869e8a5e:::benchmark_data_smithi039_455948_object167:head# 2024-09-16T17:22:32.214 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.222 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:872ef1cc:::benchmark_data_smithi039_455948_object195:head# 2024-09-16T17:22:32.222 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.231 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8923ed3b:::benchmark_data_smithi039_455948_object5:head# 2024-09-16T17:22:32.231 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.239 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8a3ec687:::benchmark_data_smithi039_455948_object214:head# 2024-09-16T17:22:32.239 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.247 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8aec744d:::benchmark_data_smithi039_455948_object254:head# 2024-09-16T17:22:32.247 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.255 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8c74c0d6:::benchmark_data_smithi039_455948_object116:head# 2024-09-16T17:22:32.256 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.264 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8eb4657a:::benchmark_data_smithi039_455948_object163:head# 2024-09-16T17:22:32.264 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.272 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92b8125e:::benchmark_data_smithi039_455948_object165:head# 2024-09-16T17:22:32.272 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.281 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92dd79ef:::benchmark_data_smithi039_455948_object130:head# 2024-09-16T17:22:32.281 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.289 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92ec70a0:::benchmark_data_smithi039_455948_object170:head# 2024-09-16T17:22:32.289 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.297 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92f0e148:::benchmark_data_smithi039_455948_object223:head# 2024-09-16T17:22:32.297 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.306 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:930eb7ac:::benchmark_data_smithi039_455948_object262:head# 2024-09-16T17:22:32.306 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.314 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:972245e1:::benchmark_data_smithi039_455948_object7:head# 2024-09-16T17:22:32.314 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.322 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:972cbf49:::benchmark_data_smithi039_455948_object3:head# 2024-09-16T17:22:32.322 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.331 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:984c6070:::benchmark_data_smithi039_455948_object1:head# 2024-09-16T17:22:32.331 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.339 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:98946638:::benchmark_data_smithi039_455948_object219:head# 2024-09-16T17:22:32.339 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.347 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:98a59f74:::benchmark_data_smithi039_455948_object216:head# 2024-09-16T17:22:32.347 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.356 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:98b0eb5d:::benchmark_data_smithi039_455948_object131:head# 2024-09-16T17:22:32.356 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.364 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:99654740:::benchmark_data_smithi039_455948_object141:head# 2024-09-16T17:22:32.364 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.373 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9b110e6f:::benchmark_data_smithi039_455948_object55:head# 2024-09-16T17:22:32.373 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.381 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9c86eb42:::benchmark_data_smithi039_455948_object287:head# 2024-09-16T17:22:32.381 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.389 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9c901805:::benchmark_data_smithi039_455948_object59:head# 2024-09-16T17:22:32.389 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.397 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9d050ed9:::benchmark_data_smithi039_455948_object284:head# 2024-09-16T17:22:32.397 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.406 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9e116042:::benchmark_data_smithi039_455948_object286:head# 2024-09-16T17:22:32.406 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.414 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9eac4879:::benchmark_data_smithi039_455948_object289:head# 2024-09-16T17:22:32.414 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.423 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9f0bc29c:::benchmark_data_smithi039_455948_object257:head# 2024-09-16T17:22:32.423 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.431 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a0acf962:::benchmark_data_smithi039_455948_object61:head# 2024-09-16T17:22:32.431 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.439 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a0b1ff33:::benchmark_data_smithi039_455948_object133:head# 2024-09-16T17:22:32.439 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.448 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a299f295:::benchmark_data_smithi039_455948_object176:head# 2024-09-16T17:22:32.448 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.456 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a3610da1:::benchmark_data_smithi039_455948_object149:head# 2024-09-16T17:22:32.456 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.465 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a3616c00:::benchmark_data_smithi039_455948_object246:head# 2024-09-16T17:22:32.465 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.473 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a3d8a56b:::benchmark_data_smithi039_455948_object73:head# 2024-09-16T17:22:32.473 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.481 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a437720d:::benchmark_data_smithi039_455948_object13:head# 2024-09-16T17:22:32.481 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.489 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5276dd6:::benchmark_data_smithi039_455948_object2:head# 2024-09-16T17:22:32.490 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.498 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5e30e12:::benchmark_data_smithi039_455948_object17:head# 2024-09-16T17:22:32.498 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.506 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5e964e3:::benchmark_data_smithi039_455948_object45:head# 2024-09-16T17:22:32.506 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.515 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5f6a828:::benchmark_data_smithi039_455948_object25:head# 2024-09-16T17:22:32.515 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.523 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a63079ff:::benchmark_data_smithi039_455948_object265:head# 2024-09-16T17:22:32.523 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.531 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a64c1b86:::benchmark_data_smithi039_455948_object288:head# 2024-09-16T17:22:32.531 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.539 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a6698c47:::benchmark_data_smithi039_455948_object259:head# 2024-09-16T17:22:32.540 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.548 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a69d0e24:::benchmark_data_smithi039_455948_object113:head# 2024-09-16T17:22:32.548 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.556 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a6c2f07b:::benchmark_data_smithi039_455948_object212:head# 2024-09-16T17:22:32.556 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.564 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a8211e63:::benchmark_data_smithi039_455948_object281:head# 2024-09-16T17:22:32.565 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.573 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ad13fc26:::benchmark_data_smithi039_455948_object110:head# 2024-09-16T17:22:32.573 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.581 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ad8239d1:::benchmark_data_smithi039_455948_object232:head# 2024-09-16T17:22:32.581 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.589 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ad91945e:::benchmark_data_smithi039_455948_object93:head# 2024-09-16T17:22:32.590 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.598 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:adaffd31:::benchmark_data_smithi039_455948_object26:head# 2024-09-16T17:22:32.598 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.606 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aeb14177:::benchmark_data_smithi039_455948_object29:head# 2024-09-16T17:22:32.606 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.615 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aecd8d89:::benchmark_data_smithi039_455948_object174:head# 2024-09-16T17:22:32.615 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.623 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b03ef840:::benchmark_data_smithi039_455948_object295:head# 2024-09-16T17:22:32.623 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.631 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b0c8cac8:::benchmark_data_smithi039_455948_object117:head# 2024-09-16T17:22:32.631 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.640 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b137eb01:::benchmark_data_smithi039_455948_object211:head# 2024-09-16T17:22:32.640 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.648 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b2347b11:::benchmark_data_smithi039_455948_object301:head# 2024-09-16T17:22:32.648 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.656 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b249856d:::benchmark_data_smithi039_455948_object235:head# 2024-09-16T17:22:32.657 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.665 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b3253c9c:::benchmark_data_smithi039_455948_object255:head# 2024-09-16T17:22:32.665 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.673 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b5b57515:::benchmark_data_smithi039_455948_object129:head# 2024-09-16T17:22:32.673 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.682 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b6c46095:::benchmark_data_smithi039_455948_object178:head# 2024-09-16T17:22:32.682 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.690 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b6ce9049:::benchmark_data_smithi039_455948_object172:head# 2024-09-16T17:22:32.690 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.723 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b6cf5c8c:::benchmark_data_smithi039_455948_object231:head# 2024-09-16T17:22:32.723 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.731 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b7c8f422:::benchmark_data_smithi039_455948_object112:head# 2024-09-16T17:22:32.732 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.740 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-16T17:22:32.740 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.748 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b8b98acb:::benchmark_data_smithi039_455948_object313:head# 2024-09-16T17:22:32.748 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.757 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b9991d10:::benchmark_data_smithi039_455948_object154:head# 2024-09-16T17:22:32.757 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.765 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:baf26fd7:::benchmark_data_smithi039_455948_object134:head# 2024-09-16T17:22:32.765 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.773 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bb71b590:::benchmark_data_smithi039_455948_object314:head# 2024-09-16T17:22:32.773 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.782 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bc052407:::benchmark_data_smithi039_455948_object270:head# 2024-09-16T17:22:32.782 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.790 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bcff6325:::benchmark_data_smithi039_455948_object315:head# 2024-09-16T17:22:32.790 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.798 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bd19ab08:::benchmark_data_smithi039_455948_object81:head# 2024-09-16T17:22:32.798 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.807 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:be7390f5:::benchmark_data_smithi039_455948_object83:head# 2024-09-16T17:22:32.807 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.815 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c07324f3:::benchmark_data_smithi039_455948_object302:head# 2024-09-16T17:22:32.815 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.823 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c0d972fb:::benchmark_data_smithi039_455948_object209:head# 2024-09-16T17:22:32.824 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.832 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c34a47c1:::benchmark_data_smithi039_455948_object319:head# 2024-09-16T17:22:32.832 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.840 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c38689dc:::benchmark_data_smithi039_455948_object50:head# 2024-09-16T17:22:32.840 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.848 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c3893c52:::benchmark_data_smithi039_455948_object66:head# 2024-09-16T17:22:32.849 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.857 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c464c3ce:::benchmark_data_smithi039_455948_object106:head# 2024-09-16T17:22:32.857 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.865 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c56ee08e:::benchmark_data_smithi039_455948_object125:head# 2024-09-16T17:22:32.865 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.873 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c6555459:::benchmark_data_smithi039_455948_object282:head# 2024-09-16T17:22:32.874 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.882 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c70cd8e6:::benchmark_data_smithi039_455948_object181:head# 2024-09-16T17:22:32.882 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.890 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c72b9798:::benchmark_data_smithi039_455948_object186:head# 2024-09-16T17:22:32.890 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.899 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c7547bd4:::benchmark_data_smithi039_455948_object160:head# 2024-09-16T17:22:32.899 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.907 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c8d50a66:::benchmark_data_smithi039_455948_object15:head# 2024-09-16T17:22:32.907 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.915 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c916bf36:::benchmark_data_smithi039_455948_object300:head# 2024-09-16T17:22:32.915 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.924 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c98347ce:::benchmark_data_smithi039_455948_object44:head# 2024-09-16T17:22:32.924 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.932 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ca283ebf:::benchmark_data_smithi039_455948_object245:head# 2024-09-16T17:22:32.932 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.940 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cb190b2d:::benchmark_data_smithi039_455948_object299:head# 2024-09-16T17:22:32.940 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.949 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cb44df0b:::benchmark_data_smithi039_455948_object102:head# 2024-09-16T17:22:32.949 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.957 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cb8dc5ee:::benchmark_data_smithi039_455948_object201:head# 2024-09-16T17:22:32.957 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.965 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cd24ea32:::benchmark_data_smithi039_455948_object42:head# 2024-09-16T17:22:32.965 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.974 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cd94f270:::benchmark_data_smithi039_455948_object124:head# 2024-09-16T17:22:32.974 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.982 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ce974f23:::benchmark_data_smithi039_455948_object159:head# 2024-09-16T17:22:32.982 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.990 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cf1dabff:::benchmark_data_smithi039_455948_object151:head# 2024-09-16T17:22:32.991 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:32.999 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cf56f22e:::benchmark_data_smithi039_455948_object294:head# 2024-09-16T17:22:32.999 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.007 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d04eceaf:::benchmark_data_smithi039_455948_object39:head# 2024-09-16T17:22:33.007 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.015 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d06d3616:::benchmark_data_smithi039_455948_object24:head# 2024-09-16T17:22:33.016 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.024 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d08457ea:::benchmark_data_smithi039_455948_object4:head# 2024-09-16T17:22:33.024 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.032 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d0851a13:::benchmark_data_smithi039_455948_object306:head# 2024-09-16T17:22:33.033 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.041 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d08dfc7d:::benchmark_data_smithi039_455948_object89:head# 2024-09-16T17:22:33.041 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.049 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d1c0a25d:::benchmark_data_smithi039_455948_object105:head# 2024-09-16T17:22:33.049 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.057 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d2347595:::benchmark_data_smithi039_455948_object123:head# 2024-09-16T17:22:33.058 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.066 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d2f20841:::benchmark_data_smithi039_455948_object95:head# 2024-09-16T17:22:33.066 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.074 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d33b721d:::benchmark_data_smithi039_455948_object244:head# 2024-09-16T17:22:33.074 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.082 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d37b01a7:::benchmark_data_smithi039_455948_object243:head# 2024-09-16T17:22:33.082 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.090 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d432d0ae:::benchmark_data_smithi039_455948_object99:head# 2024-09-16T17:22:33.091 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.099 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d4cfa604:::benchmark_data_smithi039_455948_object14:head# 2024-09-16T17:22:33.099 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.107 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d66a0831:::benchmark_data_smithi039_455948_object143:head# 2024-09-16T17:22:33.107 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.116 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d7dff5fe:::benchmark_data_smithi039_455948_object51:head# 2024-09-16T17:22:33.116 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.124 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d81d2707:::benchmark_data_smithi039_455948_object248:head# 2024-09-16T17:22:33.124 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.132 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d8cdeb85:::benchmark_data_smithi039_455948_object279:head# 2024-09-16T17:22:33.133 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.141 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d95fa43b:::benchmark_data_smithi039_455948_object97:head# 2024-09-16T17:22:33.141 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.149 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:db0a03c3:::benchmark_data_smithi039_455948_object35:head# 2024-09-16T17:22:33.149 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.158 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dbe2ac53:::benchmark_data_smithi039_455948_object92:head# 2024-09-16T17:22:33.158 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.166 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e05e2d0e:::benchmark_data_smithi039_455948_object127:head# 2024-09-16T17:22:33.166 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.192 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e18f9e35:::benchmark_data_smithi039_455948_object82:head# 2024-09-16T17:22:33.193 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.234 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e1a4e1b7:::benchmark_data_smithi039_455948_object189:head# 2024-09-16T17:22:33.234 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.257 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e2081651:::benchmark_data_smithi039_455948_object303:head# 2024-09-16T17:22:33.257 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.266 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e36479d3:::benchmark_data_smithi039_455948_object277:head# 2024-09-16T17:22:33.266 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.274 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e4509956:::benchmark_data_smithi039_455948_object222:head# 2024-09-16T17:22:33.274 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.283 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e45ad9a8:::benchmark_data_smithi039_455948_object310:head# 2024-09-16T17:22:33.283 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.291 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e49f1f25:::benchmark_data_smithi039_455948_object60:head# 2024-09-16T17:22:33.291 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.299 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e8279365:::benchmark_data_smithi039_455948_object162:head# 2024-09-16T17:22:33.299 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.308 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e84b1af3:::benchmark_data_smithi039_455948_object168:head# 2024-09-16T17:22:33.308 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.316 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e860450e:::benchmark_data_smithi039_455948_object40:head# 2024-09-16T17:22:33.316 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.393 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e9546805:::benchmark_data_smithi039_455948_object84:head# 2024-09-16T17:22:33.393 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.449 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e9c31f48:::benchmark_data_smithi039_455948_object57:head# 2024-09-16T17:22:33.450 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.517 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e9db3ae8:::benchmark_data_smithi039_455948_object185:head# 2024-09-16T17:22:33.517 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.524 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ea00b668:::benchmark_data_smithi039_455948_object126:head# 2024-09-16T17:22:33.524 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.533 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:eac552d0:::benchmark_data_smithi039_455948_object217:head# 2024-09-16T17:22:33.533 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.541 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ed7c17e0:::benchmark_data_smithi039_455948_object152:head# 2024-09-16T17:22:33.541 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.549 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:edc14e53:::benchmark_data_smithi039_455948_object230:head# 2024-09-16T17:22:33.549 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.558 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:efd04b1c:::benchmark_data_smithi039_455948_object225:head# 2024-09-16T17:22:33.558 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.566 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:efea2d35:::benchmark_data_smithi039_455948_object146:head# 2024-09-16T17:22:33.566 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.575 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f1bb07dd:::benchmark_data_smithi039_455948_object298:head# 2024-09-16T17:22:33.575 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.583 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f1e99f50:::benchmark_data_smithi039_455948_object215:head# 2024-09-16T17:22:33.583 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.591 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f1f5e565:::benchmark_data_smithi039_455948_object307:head# 2024-09-16T17:22:33.591 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.600 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f20c71ff:::benchmark_data_smithi039_455948_object267:head# 2024-09-16T17:22:33.600 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.608 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f274efb2:::benchmark_data_smithi039_455948_object27:head# 2024-09-16T17:22:33.608 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.616 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f2a8576b:::benchmark_data_smithi039_455948_object54:head# 2024-09-16T17:22:33.617 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.633 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f2f0a09c:::benchmark_data_smithi039_455948_object227:head# 2024-09-16T17:22:33.633 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.641 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f5996306:::benchmark_data_smithi039_455948_object22:head# 2024-09-16T17:22:33.642 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.650 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f61ef65d:::benchmark_data_smithi039_455948_object111:head# 2024-09-16T17:22:33.650 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.658 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f621bc7f:::benchmark_data_smithi039_455948_object182:head# 2024-09-16T17:22:33.658 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.667 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f72c3135:::benchmark_data_smithi039_455948_object21:head# 2024-09-16T17:22:33.667 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.675 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f7b30eee:::benchmark_data_smithi039_455948_object194:head# 2024-09-16T17:22:33.675 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.683 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f97e448b:::benchmark_data_smithi039_455948_object10:head# 2024-09-16T17:22:33.683 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.692 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f9d8cd97:::benchmark_data_smithi039_455948_object199:head# 2024-09-16T17:22:33.692 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.700 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f9e3593c:::benchmark_data_smithi039_455948_object256:head# 2024-09-16T17:22:33.700 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.709 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fa9bf67e:::benchmark_data_smithi039_455948_object32:head# 2024-09-16T17:22:33.709 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.717 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fadac265:::benchmark_data_smithi039_455948_object305:head# 2024-09-16T17:22:33.717 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.725 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fadc5b0b:::benchmark_data_smithi039_455948_object104:head# 2024-09-16T17:22:33.725 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.733 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fd05df8b:::benchmark_data_smithi039_455948_object70:head# 2024-09-16T17:22:33.734 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.742 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fdea465d:::benchmark_data_smithi039_455948_object107:head# 2024-09-16T17:22:33.742 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.750 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fe001570:::benchmark_data_smithi039_455948_object171:head# 2024-09-16T17:22:33.750 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.759 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ff9caaec:::benchmark_data_smithi039_455948_object263:head# 2024-09-16T17:22:33.759 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:33.767 INFO:tasks.workunit.client.0.smithi039.stdout:write_pg epoch 9 info 1.1( v 9'160 (0'0,9'160] local-lis/les=7/8 n=160 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-16T17:22:33.775 INFO:tasks.workunit.client.0.smithi039.stdout:Import successful 2024-09-16T17:22:34.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:148: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-09-16T17:22:34.970 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-16T17:22:34.970 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.0 2024-09-16T17:22:34.970 INFO:tasks.workunit.client.0.smithi039.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-16T17:22:34.971 INFO:tasks.workunit.client.0.smithi039.stdout:pool 1 pg_num 2 -> 1 2024-09-16T17:22:34.971 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:00456576:::benchmark_data_smithi039_455948_object157:head# 2024-09-16T17:22:34.971 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.009 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:014ebcb1:::benchmark_data_smithi039_455948_object233:head# 2024-09-16T17:22:35.009 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.034 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:01dc36eb:::benchmark_data_smithi039_455948_object64:head# 2024-09-16T17:22:35.034 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.059 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:01f0fd22:::benchmark_data_smithi039_455948_object208:head# 2024-09-16T17:22:35.059 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.078 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:01f5a6f9:::benchmark_data_smithi039_455948_object272:head# 2024-09-16T17:22:35.078 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.101 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:02207eee:::benchmark_data_smithi039_455948_object192:head# 2024-09-16T17:22:35.101 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.126 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:02bddb95:::benchmark_data_smithi039_455948_object137:head# 2024-09-16T17:22:35.126 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.151 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:033989ba:::benchmark_data_smithi039_455948_object80:head# 2024-09-16T17:22:35.151 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.176 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:041f4598:::benchmark_data_smithi039_455948_object187:head# 2024-09-16T17:22:35.176 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.235 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:047a0c2d:::benchmark_data_smithi039_455948_object293:head# 2024-09-16T17:22:35.235 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.269 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:04aae244:::benchmark_data_smithi039_455948_object98:head# 2024-09-16T17:22:35.269 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.326 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:058f836c:::benchmark_data_smithi039_455948_object205:head# 2024-09-16T17:22:35.326 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.351 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:05b9dc7f:::benchmark_data_smithi039_455948_object184:head# 2024-09-16T17:22:35.351 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.427 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:05f32019:::benchmark_data_smithi039_455948_object180:head# 2024-09-16T17:22:35.427 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.477 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0a2f45e2:::benchmark_data_smithi039_455948_object136:head# 2024-09-16T17:22:35.477 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.518 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0a3ddd05:::benchmark_data_smithi039_455948_object229:head# 2024-09-16T17:22:35.518 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.544 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0dc71e56:::benchmark_data_smithi039_455948_object224:head# 2024-09-16T17:22:35.544 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.568 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0e79e577:::benchmark_data_smithi039_455948_object308:head# 2024-09-16T17:22:35.568 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.593 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0e80050a:::benchmark_data_smithi039_455948_object9:head# 2024-09-16T17:22:35.593 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.610 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0f01eefd:::benchmark_data_smithi039_455948_object166:head# 2024-09-16T17:22:35.610 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.626 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0fcc09dc:::benchmark_data_smithi039_455948_object175:head# 2024-09-16T17:22:35.626 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.643 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:100e9870:::benchmark_data_smithi039_455948_object122:head# 2024-09-16T17:22:35.643 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.660 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:12727e65:::benchmark_data_smithi039_455948_object164:head# 2024-09-16T17:22:35.660 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.677 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:12810ad7:::benchmark_data_smithi039_455948_object132:head# 2024-09-16T17:22:35.677 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.693 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:12cae338:::benchmark_data_smithi039_455948_object213:head# 2024-09-16T17:22:35.693 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.710 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:13b7075e:::benchmark_data_smithi039_455948_object292:head# 2024-09-16T17:22:35.710 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.727 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:14072aff:::benchmark_data_smithi039_455948_object266:head# 2024-09-16T17:22:35.727 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.743 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:14a5c78e:::benchmark_data_smithi039_455948_object46:head# 2024-09-16T17:22:35.743 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.785 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1561ff76:::benchmark_data_smithi039_455948_object155:head# 2024-09-16T17:22:35.785 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.802 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:16d84de3:::benchmark_data_smithi039_455948_object200:head# 2024-09-16T17:22:35.802 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.818 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1a3cf39c:::benchmark_data_smithi039_455948_object177:head# 2024-09-16T17:22:35.818 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.835 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1ae97a14:::benchmark_data_smithi039_455948_object69:head# 2024-09-16T17:22:35.835 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.852 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1df9e5c8:::benchmark_data_smithi039_455948_object115:head# 2024-09-16T17:22:35.852 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.869 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1e6aee90:::benchmark_data_smithi039_455948_object312:head# 2024-09-16T17:22:35.869 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.885 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1e9dda24:::benchmark_data_smithi039_455948_object173:head# 2024-09-16T17:22:35.885 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.902 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1ea64c7e:::benchmark_data_smithi039_455948_object278:head# 2024-09-16T17:22:35.902 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.919 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1fb0f211:::benchmark_data_smithi039_455948_object109:head# 2024-09-16T17:22:35.919 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.935 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1fbf7b6c:::benchmark_data_smithi039_455948_object207:head# 2024-09-16T17:22:35.935 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.952 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1fd4abd6:::benchmark_data_smithi039_455948_object142:head# 2024-09-16T17:22:35.952 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.968 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:20997167:::benchmark_data_smithi039_455948_object56:head# 2024-09-16T17:22:35.969 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:35.985 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:227f4aa5:::benchmark_data_smithi039_455948_object260:head# 2024-09-16T17:22:35.985 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.002 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:241f88ec:::benchmark_data_smithi039_455948_object118:head# 2024-09-16T17:22:36.002 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.019 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2464192d:::benchmark_data_smithi039_455948_object67:head# 2024-09-16T17:22:36.019 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.035 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:255d9356:::benchmark_data_smithi039_455948_object19:head# 2024-09-16T17:22:36.035 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.102 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:260b2bbc:::benchmark_data_smithi039_455948_object79:head# 2024-09-16T17:22:36.102 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.119 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:27914c55:::benchmark_data_smithi039_455948_object135:head# 2024-09-16T17:22:36.119 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:279e6bbf:::benchmark_data_smithi039_455948_object247:head# 2024-09-16T17:22:36.135 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.152 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:28d56bea:::benchmark_data_smithi039_455948_object11:head# 2024-09-16T17:22:36.152 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.169 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:294aa246:::benchmark_data_smithi039_455948_object38:head# 2024-09-16T17:22:36.169 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.185 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2a8d03cc:::benchmark_data_smithi039_455948_object197:head# 2024-09-16T17:22:36.185 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.202 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2cdc5a61:::benchmark_data_smithi039_455948_object87:head# 2024-09-16T17:22:36.202 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.219 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2f2af160:::benchmark_data_smithi039_455948_object63:head# 2024-09-16T17:22:36.219 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.235 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2f6ad66d:::benchmark_data_smithi039_455948_object237:head# 2024-09-16T17:22:36.235 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2fa31139:::benchmark_data_smithi039_455948_object36:head# 2024-09-16T17:22:36.252 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.269 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2faa5036:::benchmark_data_smithi039_455948_object43:head# 2024-09-16T17:22:36.269 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.286 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3061c169:::benchmark_data_smithi039_455948_object271:head# 2024-09-16T17:22:36.286 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.302 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:30da3c6c:::benchmark_data_smithi039_455948_object193:head# 2024-09-16T17:22:36.302 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.319 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3105728e:::benchmark_data_smithi039_455948_object148:head# 2024-09-16T17:22:36.319 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.336 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3173dfe0:::benchmark_data_smithi039_455948_object188:head# 2024-09-16T17:22:36.336 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.352 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:335c1a68:::benchmark_data_smithi039_455948_object128:head# 2024-09-16T17:22:36.352 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.369 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:351c3af9:::benchmark_data_smithi039_455948_object72:head# 2024-09-16T17:22:36.369 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.385 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:36bb23c0:::benchmark_data_smithi039_455948_object101:head# 2024-09-16T17:22:36.386 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.402 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:375cb5d3:::benchmark_data_smithi039_455948_object275:head# 2024-09-16T17:22:36.402 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.419 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:37a5f014:::benchmark_data_smithi039_455948_object204:head# 2024-09-16T17:22:36.419 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.435 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3a6cafe9:::benchmark_data_smithi039_455948_object6:head# 2024-09-16T17:22:36.436 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.477 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3c81aa17:::benchmark_data_smithi039_455948_object280:head# 2024-09-16T17:22:36.477 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.494 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3cd1f324:::benchmark_data_smithi039_455948_object202:head# 2024-09-16T17:22:36.494 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.511 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3eaccc36:::benchmark_data_smithi039_455948_object144:head# 2024-09-16T17:22:36.511 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.527 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3f37ec22:::benchmark_data_smithi039_455948_object161:head# 2024-09-16T17:22:36.528 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.544 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3fc02a40:::benchmark_data_smithi039_455948_object74:head# 2024-09-16T17:22:36.544 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.561 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4003301f:::benchmark_data_smithi039_455948_object139:head# 2024-09-16T17:22:36.561 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.577 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4117f431:::benchmark_data_smithi039_455948_object234:head# 2024-09-16T17:22:36.577 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.594 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:41facc71:::benchmark_data_smithi039_455948_object198:head# 2024-09-16T17:22:36.594 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.611 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4411f440:::benchmark_data_smithi039_455948_object297:head# 2024-09-16T17:22:36.611 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.627 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:44802c6d:::benchmark_data_smithi039_455948_object119:head# 2024-09-16T17:22:36.627 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.644 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:45c521fb:::benchmark_data_smithi039_455948_object249:head# 2024-09-16T17:22:36.644 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.661 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:469ace3a:::benchmark_data_smithi039_455948_object206:head# 2024-09-16T17:22:36.661 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.672 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:47970e62:::benchmark_data_smithi039_455948_object210:head# 2024-09-16T17:22:36.672 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.677 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:47b13862:::benchmark_data_smithi039_455948_object138:head# 2024-09-16T17:22:36.678 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.694 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4801d23d:::benchmark_data_smithi039_455948_object85:head# 2024-09-16T17:22:36.695 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.711 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:491895da:::benchmark_data_smithi039_455948_object169:head# 2024-09-16T17:22:36.711 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.728 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4935d551:::benchmark_data_smithi039_455948_object251:head# 2024-09-16T17:22:36.728 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.744 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:496b18ad:::benchmark_data_smithi039_455948_object253:head# 2024-09-16T17:22:36.744 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.753 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4a184ab0:::benchmark_data_smithi039_455948_object158:head# 2024-09-16T17:22:36.753 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.761 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4a1d7637:::benchmark_data_smithi039_455948_object183:head# 2024-09-16T17:22:36.761 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.769 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4ac873c7:::benchmark_data_smithi039_455948_object37:head# 2024-09-16T17:22:36.769 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.778 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4bc4a2e0:::benchmark_data_smithi039_455948_object49:head# 2024-09-16T17:22:36.778 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.786 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4c5a7992:::benchmark_data_smithi039_455948_object221:head# 2024-09-16T17:22:36.786 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.794 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4f03ab2e:::benchmark_data_smithi039_455948_object318:head# 2024-09-16T17:22:36.795 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.803 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4f301666:::benchmark_data_smithi039_455948_object228:head# 2024-09-16T17:22:36.803 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.811 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4f3a0a7a:::benchmark_data_smithi039_455948_object240:head# 2024-09-16T17:22:36.811 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.819 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4fb8888a:::benchmark_data_smithi039_455948_object203:head# 2024-09-16T17:22:36.820 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.828 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:50a96376:::benchmark_data_smithi039_455948_object96:head# 2024-09-16T17:22:36.828 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.836 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5244e47b:::benchmark_data_smithi039_455948_object62:head# 2024-09-16T17:22:36.836 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.844 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:52d42a8e:::benchmark_data_smithi039_455948_object48:head# 2024-09-16T17:22:36.845 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.853 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:52ede5be:::benchmark_data_smithi039_455948_object12:head# 2024-09-16T17:22:36.853 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.861 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53a105b2:::benchmark_data_smithi039_455948_object311:head# 2024-09-16T17:22:36.861 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.870 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53cb9fd2:::benchmark_data_smithi039_455948_object190:head# 2024-09-16T17:22:36.870 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.878 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53ed147c:::benchmark_data_smithi039_455948_object31:head# 2024-09-16T17:22:36.878 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.886 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:54d49a7f:::benchmark_data_smithi039_455948_object28:head# 2024-09-16T17:22:36.886 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.895 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:54e04d99:::benchmark_data_smithi039_455948_object283:head# 2024-09-16T17:22:36.895 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.903 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:54e85f22:::benchmark_data_smithi039_455948_object23:head# 2024-09-16T17:22:36.903 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.911 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5651eea5:::benchmark_data_smithi039_455948_object18:head# 2024-09-16T17:22:36.912 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.920 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:57dabe70:::benchmark_data_smithi039_455948_object218:head# 2024-09-16T17:22:36.920 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.928 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5801fe81:::benchmark_data_smithi039_455948_object75:head# 2024-09-16T17:22:36.928 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.936 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:58d2c70a:::benchmark_data_smithi039_455948_object0:head# 2024-09-16T17:22:36.936 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.945 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:59614fc8:::benchmark_data_smithi039_455948_object76:head# 2024-09-16T17:22:36.945 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.953 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5a3d91e5:::benchmark_data_smithi039_455948_object108:head# 2024-09-16T17:22:36.953 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.962 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5a7ae6c2:::benchmark_data_smithi039_455948_object100:head# 2024-09-16T17:22:36.962 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:36.970 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5c28febb:::benchmark_data_smithi039_455948_object156:head# 2024-09-16T17:22:36.970 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.003 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5db907e9:::benchmark_data_smithi039_455948_object121:head# 2024-09-16T17:22:37.003 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.012 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ded13e4:::benchmark_data_smithi039_455948_object90:head# 2024-09-16T17:22:37.012 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.020 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5e428851:::benchmark_data_smithi039_455948_object309:head# 2024-09-16T17:22:37.020 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.028 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5e629290:::benchmark_data_smithi039_455948_object20:head# 2024-09-16T17:22:37.028 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.037 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5efeb4ae:::benchmark_data_smithi039_455948_object153:head# 2024-09-16T17:22:37.037 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.045 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ff59d48:::benchmark_data_smithi039_455948_object16:head# 2024-09-16T17:22:37.045 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.053 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6137bcba:::benchmark_data_smithi039_455948_object120:head# 2024-09-16T17:22:37.053 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.062 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6154b590:::benchmark_data_smithi039_455948_object58:head# 2024-09-16T17:22:37.062 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.070 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6280b741:::benchmark_data_smithi039_455948_object250:head# 2024-09-16T17:22:37.070 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.080 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:62961663:::benchmark_data_smithi039_455948_object296:head# 2024-09-16T17:22:37.080 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.087 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:642ab5c2:::benchmark_data_smithi039_455948_object41:head# 2024-09-16T17:22:37.087 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.095 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:64aa8ba8:::benchmark_data_smithi039_455948_object71:head# 2024-09-16T17:22:37.095 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.103 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:654ddc28:::benchmark_data_smithi039_455948_object147:head# 2024-09-16T17:22:37.103 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.112 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:656a01f0:::benchmark_data_smithi039_455948_object316:head# 2024-09-16T17:22:37.112 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.120 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6667fb4d:::benchmark_data_smithi039_455948_object252:head# 2024-09-16T17:22:37.120 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.129 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:66719837:::benchmark_data_smithi039_455948_object179:head# 2024-09-16T17:22:37.129 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.137 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:67fc60d1:::benchmark_data_smithi039_455948_object103:head# 2024-09-16T17:22:37.137 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.145 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:67fc91c1:::benchmark_data_smithi039_455948_object52:head# 2024-09-16T17:22:37.145 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.153 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6815ad2e:::benchmark_data_smithi039_455948_object196:head# 2024-09-16T17:22:37.153 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.228 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:69293e2b:::benchmark_data_smithi039_455948_object290:head# 2024-09-16T17:22:37.228 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.270 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6ae4f661:::benchmark_data_smithi039_455948_object241:head# 2024-09-16T17:22:37.270 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.287 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6cb11b42:::benchmark_data_smithi039_455948_object285:head# 2024-09-16T17:22:37.287 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.295 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6ce0dbf0:::benchmark_data_smithi039_455948_object268:head# 2024-09-16T17:22:37.295 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.304 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6d2fe8c5:::benchmark_data_smithi039_455948_object273:head# 2024-09-16T17:22:37.304 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.312 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6d540a4e:::benchmark_data_smithi039_455948_object8:head# 2024-09-16T17:22:37.312 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.320 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6d724163:::benchmark_data_smithi039_455948_object269:head# 2024-09-16T17:22:37.320 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.329 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6e61b9d5:::benchmark_data_smithi039_455948_object68:head# 2024-09-16T17:22:37.329 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.337 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6f095114:::benchmark_data_smithi039_455948_object94:head# 2024-09-16T17:22:37.337 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.345 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:70691fc8:::benchmark_data_smithi039_455948_object145:head# 2024-09-16T17:22:37.345 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.354 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:708034b3:::benchmark_data_smithi039_455948_object276:head# 2024-09-16T17:22:37.354 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.362 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:70f55f31:::benchmark_data_smithi039_455948_object239:head# 2024-09-16T17:22:37.362 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.370 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:71b1cb82:::benchmark_data_smithi039_455948_object65:head# 2024-09-16T17:22:37.370 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.431 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:71ce8906:::benchmark_data_smithi039_455948_object78:head# 2024-09-16T17:22:37.431 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.480 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:72fdddd5:::benchmark_data_smithi039_455948_object86:head# 2024-09-16T17:22:37.480 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.537 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:73b633f2:::benchmark_data_smithi039_455948_object150:head# 2024-09-16T17:22:37.537 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.545 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:754b4658:::benchmark_data_smithi039_455948_object77:head# 2024-09-16T17:22:37.545 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.554 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:75a63f9f:::benchmark_data_smithi039_455948_object226:head# 2024-09-16T17:22:37.554 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.562 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:75f9bf67:::benchmark_data_smithi039_455948_object242:head# 2024-09-16T17:22:37.562 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.570 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:764ab525:::benchmark_data_smithi039_455948_object317:head# 2024-09-16T17:22:37.571 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.579 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:768e2761:::benchmark_data_smithi039_455948_object238:head# 2024-09-16T17:22:37.579 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.587 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:77182e0f:::benchmark_data_smithi039_455948_object140:head# 2024-09-16T17:22:37.587 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.596 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7739f249:::benchmark_data_smithi039_455948_object34:head# 2024-09-16T17:22:37.596 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.604 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:784592ec:::benchmark_data_smithi039_455948_object33:head# 2024-09-16T17:22:37.604 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.612 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7a597a33:::benchmark_data_smithi039_455948_object274:head# 2024-09-16T17:22:37.612 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.621 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7b291d50:::benchmark_data_smithi039_455948_object220:head# 2024-09-16T17:22:37.621 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.629 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7c07aaad:::benchmark_data_smithi039_455948_object91:head# 2024-09-16T17:22:37.629 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.637 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7cd2d272:::benchmark_data_smithi039_455948_object53:head# 2024-09-16T17:22:37.637 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.646 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7d0d8840:::benchmark_data_smithi039_455948_object264:head# 2024-09-16T17:22:37.646 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.654 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7d342ae1:::benchmark_data_smithi039_455948_object47:head# 2024-09-16T17:22:37.654 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.662 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7ee172f3:::benchmark_data_smithi039_455948_object304:head# 2024-09-16T17:22:37.662 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.671 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7fe7c24d:::benchmark_data_smithi039_455948_object88:head# 2024-09-16T17:22:37.671 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:37.679 INFO:tasks.workunit.client.0.smithi039.stdout:write_pg epoch 9 info 1.0( v 9'161 (0'0,9'161] local-lis/les=7/8 n=161 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-16T17:22:37.696 INFO:tasks.workunit.client.0.smithi039.stdout:Import successful 2024-09-16T17:22:38.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:151: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-09-16T17:22:38.883 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-16T17:22:38.884 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-16T17:22:38.884 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-16T17:22:39.013 INFO:tasks.workunit.client.0.smithi039.stdout:Remove successful 2024-09-16T17:22:39.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:152: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-09-16T17:22:40.191 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-16T17:22:40.192 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-16T17:22:40.192 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-09-16T17:22:40.271 INFO:tasks.workunit.client.0.smithi039.stdout:Remove successful 2024-09-16T17:22:40.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:154: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0.later --force 2024-09-16T17:22:41.515 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-16T17:22:41.515 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.0 2024-09-16T17:22:41.516 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:00456576:::benchmark_data_smithi039_455948_object157:head# 2024-09-16T17:22:41.516 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.580 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:00eb4400:::benchmark_data_smithi039_461800_object119:head# 2024-09-16T17:22:41.580 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.605 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:014ebcb1:::benchmark_data_smithi039_455948_object233:head# 2024-09-16T17:22:41.605 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.630 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:01dc36eb:::benchmark_data_smithi039_455948_object64:head# 2024-09-16T17:22:41.630 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.680 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:01f0fd22:::benchmark_data_smithi039_455948_object208:head# 2024-09-16T17:22:41.680 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.697 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:01f5a6f9:::benchmark_data_smithi039_455948_object272:head# 2024-09-16T17:22:41.697 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.705 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:02207eee:::benchmark_data_smithi039_455948_object192:head# 2024-09-16T17:22:41.705 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.714 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:02bddb95:::benchmark_data_smithi039_455948_object137:head# 2024-09-16T17:22:41.714 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.722 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:033989ba:::benchmark_data_smithi039_455948_object80:head# 2024-09-16T17:22:41.722 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.730 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:041f4598:::benchmark_data_smithi039_455948_object187:head# 2024-09-16T17:22:41.730 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.739 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:047a0c2d:::benchmark_data_smithi039_455948_object293:head# 2024-09-16T17:22:41.739 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.747 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:048eb4a3:::benchmark_data_smithi039_461800_object10:head# 2024-09-16T17:22:41.747 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.755 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:04aae244:::benchmark_data_smithi039_455948_object98:head# 2024-09-16T17:22:41.755 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.764 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:058f836c:::benchmark_data_smithi039_455948_object205:head# 2024-09-16T17:22:41.764 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.772 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:05b9dc7f:::benchmark_data_smithi039_455948_object184:head# 2024-09-16T17:22:41.772 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.780 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:05f32019:::benchmark_data_smithi039_455948_object180:head# 2024-09-16T17:22:41.780 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.789 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:06327336:::benchmark_data_smithi039_461800_object122:head# 2024-09-16T17:22:41.789 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.797 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:064aeecb:::benchmark_data_smithi039_461800_object158:head# 2024-09-16T17:22:41.797 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.805 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0681ed1a:::benchmark_data_smithi039_461800_object170:head# 2024-09-16T17:22:41.805 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.814 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0749227f:::benchmark_data_smithi039_461800_object50:head# 2024-09-16T17:22:41.814 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.822 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:07512169:::benchmark_data_smithi039_461800_object77:head# 2024-09-16T17:22:41.822 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.830 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:07fcaf55:::benchmark_data_smithi039_461800_object45:head# 2024-09-16T17:22:41.830 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.839 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0a2f45e2:::benchmark_data_smithi039_455948_object136:head# 2024-09-16T17:22:41.839 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.847 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0a3ddd05:::benchmark_data_smithi039_455948_object229:head# 2024-09-16T17:22:41.847 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.855 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0d9d2be8:::benchmark_data_smithi039_461800_object101:head# 2024-09-16T17:22:41.855 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.864 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0dc71e56:::benchmark_data_smithi039_455948_object224:head# 2024-09-16T17:22:41.864 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.872 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0e128744:::benchmark_data_smithi039_461800_object42:head# 2024-09-16T17:22:41.872 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.880 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0e79e577:::benchmark_data_smithi039_455948_object308:head# 2024-09-16T17:22:41.881 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.889 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0e80050a:::benchmark_data_smithi039_455948_object9:head# 2024-09-16T17:22:41.889 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.897 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0f01eefd:::benchmark_data_smithi039_455948_object166:head# 2024-09-16T17:22:41.897 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.905 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0fcc09dc:::benchmark_data_smithi039_455948_object175:head# 2024-09-16T17:22:41.905 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.914 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:100e9870:::benchmark_data_smithi039_455948_object122:head# 2024-09-16T17:22:41.914 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.922 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1060baad:::benchmark_data_smithi039_461800_object113:head# 2024-09-16T17:22:41.922 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.930 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1074fdbd:::benchmark_data_smithi039_461800_object115:head# 2024-09-16T17:22:41.931 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.939 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:12716124:::benchmark_data_smithi039_461800_object85:head# 2024-09-16T17:22:41.939 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.947 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:12727e65:::benchmark_data_smithi039_455948_object164:head# 2024-09-16T17:22:41.947 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.955 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:12810ad7:::benchmark_data_smithi039_455948_object132:head# 2024-09-16T17:22:41.956 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.964 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:12cae338:::benchmark_data_smithi039_455948_object213:head# 2024-09-16T17:22:41.964 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.972 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1330884b:::benchmark_data_smithi039_461800_object68:head# 2024-09-16T17:22:41.972 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.981 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:138e2f90:::benchmark_data_smithi039_461800_object43:head# 2024-09-16T17:22:41.981 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.989 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:13b7075e:::benchmark_data_smithi039_455948_object292:head# 2024-09-16T17:22:41.989 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:41.997 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:14072aff:::benchmark_data_smithi039_455948_object266:head# 2024-09-16T17:22:41.997 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.006 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:14a5c78e:::benchmark_data_smithi039_455948_object46:head# 2024-09-16T17:22:42.006 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.014 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1561ff76:::benchmark_data_smithi039_455948_object155:head# 2024-09-16T17:22:42.014 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.022 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:15a4150c:::benchmark_data_smithi039_461800_object69:head# 2024-09-16T17:22:42.022 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.031 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:16d84de3:::benchmark_data_smithi039_455948_object200:head# 2024-09-16T17:22:42.031 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.039 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:19661353:::benchmark_data_smithi039_461800_object51:head# 2024-09-16T17:22:42.039 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.047 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1a3cf39c:::benchmark_data_smithi039_455948_object177:head# 2024-09-16T17:22:42.047 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.056 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1ae97a14:::benchmark_data_smithi039_455948_object69:head# 2024-09-16T17:22:42.056 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.064 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1b1fa822:::benchmark_data_smithi039_461800_object143:head# 2024-09-16T17:22:42.064 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.072 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1bcacdd1:::benchmark_data_smithi039_461800_object98:head# 2024-09-16T17:22:42.073 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.081 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1dda2954:::benchmark_data_smithi039_461800_object33:head# 2024-09-16T17:22:42.081 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.089 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1df9e5c8:::benchmark_data_smithi039_455948_object115:head# 2024-09-16T17:22:42.089 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.097 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1e34d247:::benchmark_data_smithi039_461800_object92:head# 2024-09-16T17:22:42.098 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.106 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1e6aee90:::benchmark_data_smithi039_455948_object312:head# 2024-09-16T17:22:42.106 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.114 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1e9dda24:::benchmark_data_smithi039_455948_object173:head# 2024-09-16T17:22:42.114 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.123 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1ea64c7e:::benchmark_data_smithi039_455948_object278:head# 2024-09-16T17:22:42.123 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.131 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1ebea92e:::benchmark_data_smithi039_461800_object175:head# 2024-09-16T17:22:42.131 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.139 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1fb0f211:::benchmark_data_smithi039_455948_object109:head# 2024-09-16T17:22:42.139 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.148 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1fbf7b6c:::benchmark_data_smithi039_455948_object207:head# 2024-09-16T17:22:42.148 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.156 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1fd4abd6:::benchmark_data_smithi039_455948_object142:head# 2024-09-16T17:22:42.156 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.164 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:208ac2b2:::benchmark_data_smithi039_461800_object130:head# 2024-09-16T17:22:42.164 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.173 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:20997167:::benchmark_data_smithi039_455948_object56:head# 2024-09-16T17:22:42.173 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.181 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:227f4aa5:::benchmark_data_smithi039_455948_object260:head# 2024-09-16T17:22:42.181 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.189 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:241b6c7b:::benchmark_data_smithi039_461800_object55:head# 2024-09-16T17:22:42.189 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.239 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:241f88ec:::benchmark_data_smithi039_455948_object118:head# 2024-09-16T17:22:42.239 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.248 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2464192d:::benchmark_data_smithi039_455948_object67:head# 2024-09-16T17:22:42.248 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.256 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:255d9356:::benchmark_data_smithi039_455948_object19:head# 2024-09-16T17:22:42.256 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.264 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:260b2bbc:::benchmark_data_smithi039_455948_object79:head# 2024-09-16T17:22:42.265 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.273 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:27914c55:::benchmark_data_smithi039_455948_object135:head# 2024-09-16T17:22:42.273 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.281 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:279e6bbf:::benchmark_data_smithi039_455948_object247:head# 2024-09-16T17:22:42.281 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.289 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:28d56bea:::benchmark_data_smithi039_455948_object11:head# 2024-09-16T17:22:42.290 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.298 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:294aa246:::benchmark_data_smithi039_455948_object38:head# 2024-09-16T17:22:42.298 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.306 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:29a8d043:::benchmark_data_smithi039_461800_object94:head# 2024-09-16T17:22:42.306 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.314 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:29d21410:::benchmark_data_smithi039_461800_object21:head# 2024-09-16T17:22:42.315 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.323 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:29d337cd:::benchmark_data_smithi039_461800_object24:head# 2024-09-16T17:22:42.323 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.331 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2a8d03cc:::benchmark_data_smithi039_455948_object197:head# 2024-09-16T17:22:42.331 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.340 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2ba0cc00:::benchmark_data_smithi039_461800_object39:head# 2024-09-16T17:22:42.340 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.348 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2c3b99e3:::benchmark_data_smithi039_461800_object108:head# 2024-09-16T17:22:42.348 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.356 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2cdc5a61:::benchmark_data_smithi039_455948_object87:head# 2024-09-16T17:22:42.356 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.365 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2de2bac8:::benchmark_data_smithi039_461800_object155:head# 2024-09-16T17:22:42.365 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.373 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2e4027e4:::benchmark_data_smithi039_461800_object70:head# 2024-09-16T17:22:42.373 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.381 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2e79016e:::benchmark_data_smithi039_461800_object8:head# 2024-09-16T17:22:42.381 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.390 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2ec5c54e:::benchmark_data_smithi039_461800_object74:head# 2024-09-16T17:22:42.390 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.398 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2f2af160:::benchmark_data_smithi039_455948_object63:head# 2024-09-16T17:22:42.398 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.406 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2f6ad66d:::benchmark_data_smithi039_455948_object237:head# 2024-09-16T17:22:42.406 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.415 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2f7d0174:::benchmark_data_smithi039_461800_object146:head# 2024-09-16T17:22:42.415 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.423 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2fa31139:::benchmark_data_smithi039_455948_object36:head# 2024-09-16T17:22:42.423 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.431 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2faa5036:::benchmark_data_smithi039_455948_object43:head# 2024-09-16T17:22:42.432 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.440 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3061c169:::benchmark_data_smithi039_455948_object271:head# 2024-09-16T17:22:42.440 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.448 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:30da3c6c:::benchmark_data_smithi039_455948_object193:head# 2024-09-16T17:22:42.448 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.457 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3105728e:::benchmark_data_smithi039_455948_object148:head# 2024-09-16T17:22:42.457 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.465 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3173dfe0:::benchmark_data_smithi039_455948_object188:head# 2024-09-16T17:22:42.465 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.473 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:334e8142:::benchmark_data_smithi039_461800_object78:head# 2024-09-16T17:22:42.473 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.482 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:335c1a68:::benchmark_data_smithi039_455948_object128:head# 2024-09-16T17:22:42.482 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.490 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:34153b00:::benchmark_data_smithi039_461800_object136:head# 2024-09-16T17:22:42.490 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.498 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:34d3b2fb:::benchmark_data_smithi039_461800_object142:head# 2024-09-16T17:22:42.498 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.507 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3509581c:::benchmark_data_smithi039_461800_object153:head# 2024-09-16T17:22:42.507 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.515 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:351c3af9:::benchmark_data_smithi039_455948_object72:head# 2024-09-16T17:22:42.515 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.523 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:369b8773:::benchmark_data_smithi039_461800_object133:head# 2024-09-16T17:22:42.523 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.532 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:36bb23c0:::benchmark_data_smithi039_455948_object101:head# 2024-09-16T17:22:42.532 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.549 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:375cb5d3:::benchmark_data_smithi039_455948_object275:head# 2024-09-16T17:22:42.549 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.557 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:37a5f014:::benchmark_data_smithi039_455948_object204:head# 2024-09-16T17:22:42.557 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.565 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:38196426:::benchmark_data_smithi039_461800_object30:head# 2024-09-16T17:22:42.565 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.574 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:39160a3b:::benchmark_data_smithi039_461800_object75:head# 2024-09-16T17:22:42.574 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.582 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3a6cafe9:::benchmark_data_smithi039_455948_object6:head# 2024-09-16T17:22:42.583 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.590 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3aab4211:::benchmark_data_smithi039_461800_object161:head# 2024-09-16T17:22:42.591 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.599 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3b836eaa:::benchmark_data_smithi039_461800_object154:head# 2024-09-16T17:22:42.599 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.607 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3c81aa17:::benchmark_data_smithi039_455948_object280:head# 2024-09-16T17:22:42.607 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.615 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3cd1f324:::benchmark_data_smithi039_455948_object202:head# 2024-09-16T17:22:42.616 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.624 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3d03a78e:::benchmark_data_smithi039_461800_object171:head# 2024-09-16T17:22:42.624 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.632 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3e761f82:::benchmark_data_smithi039_461800_object9:head# 2024-09-16T17:22:42.632 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.641 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3eaccc36:::benchmark_data_smithi039_455948_object144:head# 2024-09-16T17:22:42.641 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.649 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3f37ec22:::benchmark_data_smithi039_455948_object161:head# 2024-09-16T17:22:42.649 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.657 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3fc02a40:::benchmark_data_smithi039_455948_object74:head# 2024-09-16T17:22:42.657 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.666 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4003301f:::benchmark_data_smithi039_455948_object139:head# 2024-09-16T17:22:42.666 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.674 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4117f431:::benchmark_data_smithi039_455948_object234:head# 2024-09-16T17:22:42.674 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.682 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:41facc71:::benchmark_data_smithi039_455948_object198:head# 2024-09-16T17:22:42.682 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.691 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:430c10f6:::benchmark_data_smithi039_461800_object124:head# 2024-09-16T17:22:42.691 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.699 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4411f440:::benchmark_data_smithi039_455948_object297:head# 2024-09-16T17:22:42.699 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.707 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:44802c6d:::benchmark_data_smithi039_455948_object119:head# 2024-09-16T17:22:42.707 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.716 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:45c521fb:::benchmark_data_smithi039_455948_object249:head# 2024-09-16T17:22:42.716 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.724 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:45c56cfe:::benchmark_data_smithi039_461800_object80:head# 2024-09-16T17:22:42.724 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.732 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:469ace3a:::benchmark_data_smithi039_455948_object206:head# 2024-09-16T17:22:42.733 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.741 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:47970e62:::benchmark_data_smithi039_455948_object210:head# 2024-09-16T17:22:42.741 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.749 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:47b13862:::benchmark_data_smithi039_455948_object138:head# 2024-09-16T17:22:42.749 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.757 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4801d23d:::benchmark_data_smithi039_455948_object85:head# 2024-09-16T17:22:42.758 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.766 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4817f58f:::benchmark_data_smithi039_461800_object18:head# 2024-09-16T17:22:42.766 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.774 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4899b9aa:::benchmark_data_smithi039_461800_object79:head# 2024-09-16T17:22:42.774 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.816 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:491895da:::benchmark_data_smithi039_455948_object169:head# 2024-09-16T17:22:42.816 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.824 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:49285683:::benchmark_data_smithi039_461800_object47:head# 2024-09-16T17:22:42.824 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.833 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4935d551:::benchmark_data_smithi039_455948_object251:head# 2024-09-16T17:22:42.833 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.841 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:496b18ad:::benchmark_data_smithi039_455948_object253:head# 2024-09-16T17:22:42.841 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.849 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4a184ab0:::benchmark_data_smithi039_455948_object158:head# 2024-09-16T17:22:42.849 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.858 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4a1d7637:::benchmark_data_smithi039_455948_object183:head# 2024-09-16T17:22:42.858 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.866 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4ac873c7:::benchmark_data_smithi039_455948_object37:head# 2024-09-16T17:22:42.866 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.874 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4bc4a2e0:::benchmark_data_smithi039_455948_object49:head# 2024-09-16T17:22:42.874 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.883 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4c4cedac:::benchmark_data_smithi039_461800_object120:head# 2024-09-16T17:22:42.883 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.891 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4c531f09:::benchmark_data_smithi039_461800_object23:head# 2024-09-16T17:22:42.891 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.899 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4c5a7992:::benchmark_data_smithi039_455948_object221:head# 2024-09-16T17:22:42.899 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.908 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4c605a30:::benchmark_data_smithi039_461800_object100:head# 2024-09-16T17:22:42.908 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.916 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4d1c2c0b:::benchmark_data_smithi039_461800_object73:head# 2024-09-16T17:22:42.916 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.924 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4eaae14b:::benchmark_data_smithi039_461800_object67:head# 2024-09-16T17:22:42.925 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.933 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4f03ab2e:::benchmark_data_smithi039_455948_object318:head# 2024-09-16T17:22:42.933 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.941 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4f301666:::benchmark_data_smithi039_455948_object228:head# 2024-09-16T17:22:42.941 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.949 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4f3a0a7a:::benchmark_data_smithi039_455948_object240:head# 2024-09-16T17:22:42.950 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.958 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4fb8888a:::benchmark_data_smithi039_455948_object203:head# 2024-09-16T17:22:42.958 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.966 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:50a96376:::benchmark_data_smithi039_455948_object96:head# 2024-09-16T17:22:42.966 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.975 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:51ece5aa:::benchmark_data_smithi039_461800_object144:head# 2024-09-16T17:22:42.975 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.983 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5244e47b:::benchmark_data_smithi039_455948_object62:head# 2024-09-16T17:22:42.983 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:42.991 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:52d42a8e:::benchmark_data_smithi039_455948_object48:head# 2024-09-16T17:22:42.991 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.000 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:52e6e778:::benchmark_data_smithi039_461800_object105:head# 2024-09-16T17:22:43.000 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.008 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:52ede5be:::benchmark_data_smithi039_455948_object12:head# 2024-09-16T17:22:43.008 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.016 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53a105b2:::benchmark_data_smithi039_455948_object311:head# 2024-09-16T17:22:43.016 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.025 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53cb9fd2:::benchmark_data_smithi039_455948_object190:head# 2024-09-16T17:22:43.025 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.033 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53ed147c:::benchmark_data_smithi039_455948_object31:head# 2024-09-16T17:22:43.033 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.041 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:54d49a7f:::benchmark_data_smithi039_455948_object28:head# 2024-09-16T17:22:43.041 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.050 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:54e04d99:::benchmark_data_smithi039_455948_object283:head# 2024-09-16T17:22:43.050 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.058 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:54e85f22:::benchmark_data_smithi039_455948_object23:head# 2024-09-16T17:22:43.058 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.066 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:55cbeda0:::benchmark_data_smithi039_461800_object123:head# 2024-09-16T17:22:43.067 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.075 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5651eea5:::benchmark_data_smithi039_455948_object18:head# 2024-09-16T17:22:43.075 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.083 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:57210b16:::benchmark_data_smithi039_461800_object72:head# 2024-09-16T17:22:43.083 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.092 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:57dabe70:::benchmark_data_smithi039_455948_object218:head# 2024-09-16T17:22:43.092 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.100 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5801fe81:::benchmark_data_smithi039_455948_object75:head# 2024-09-16T17:22:43.100 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.108 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:58d2c70a:::benchmark_data_smithi039_455948_object0:head# 2024-09-16T17:22:43.108 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.117 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:59614fc8:::benchmark_data_smithi039_455948_object76:head# 2024-09-16T17:22:43.117 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.125 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:598ce77d:::benchmark_data_smithi039_461800_object28:head# 2024-09-16T17:22:43.125 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.133 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:59c6e486:::benchmark_data_smithi039_461800_object164:head# 2024-09-16T17:22:43.134 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.142 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5a3d91e5:::benchmark_data_smithi039_455948_object108:head# 2024-09-16T17:22:43.142 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.150 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5a481a4a:::benchmark_data_smithi039_461800_object88:head# 2024-09-16T17:22:43.150 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.158 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5a7ae6c2:::benchmark_data_smithi039_455948_object100:head# 2024-09-16T17:22:43.158 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.167 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5c28febb:::benchmark_data_smithi039_455948_object156:head# 2024-09-16T17:22:43.167 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.200 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5d03c722:::benchmark_data_smithi039_461800_object141:head# 2024-09-16T17:22:43.200 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.240 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5db907e9:::benchmark_data_smithi039_455948_object121:head# 2024-09-16T17:22:43.240 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.258 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ded13e4:::benchmark_data_smithi039_455948_object90:head# 2024-09-16T17:22:43.259 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.267 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5e428851:::benchmark_data_smithi039_455948_object309:head# 2024-09-16T17:22:43.267 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.275 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5e629290:::benchmark_data_smithi039_455948_object20:head# 2024-09-16T17:22:43.275 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.284 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ee4ed35:::benchmark_data_smithi039_461800_object139:head# 2024-09-16T17:22:43.284 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.292 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5efeb4ae:::benchmark_data_smithi039_455948_object153:head# 2024-09-16T17:22:43.292 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.300 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5fd55c10:::benchmark_data_smithi039_461800_object97:head# 2024-09-16T17:22:43.300 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.309 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ff59d48:::benchmark_data_smithi039_455948_object16:head# 2024-09-16T17:22:43.309 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.317 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:606af174:::benchmark_data_smithi039_461800_object148:head# 2024-09-16T17:22:43.317 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.325 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6114f880:::benchmark_data_smithi039_461800_object90:head# 2024-09-16T17:22:43.325 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.334 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6137bcba:::benchmark_data_smithi039_455948_object120:head# 2024-09-16T17:22:43.334 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.342 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6154b590:::benchmark_data_smithi039_455948_object58:head# 2024-09-16T17:22:43.342 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.350 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:61a5de33:::benchmark_data_smithi039_461800_object149:head# 2024-09-16T17:22:43.350 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.391 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6280b741:::benchmark_data_smithi039_455948_object250:head# 2024-09-16T17:22:43.391 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.417 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:62961663:::benchmark_data_smithi039_455948_object296:head# 2024-09-16T17:22:43.417 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.467 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:633b496b:::benchmark_data_smithi039_461800_object126:head# 2024-09-16T17:22:43.467 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.492 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:635c8e6d:::benchmark_data_smithi039_461800_object56:head# 2024-09-16T17:22:43.492 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.500 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:642ab5c2:::benchmark_data_smithi039_455948_object41:head# 2024-09-16T17:22:43.501 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.509 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:64aa8ba8:::benchmark_data_smithi039_455948_object71:head# 2024-09-16T17:22:43.509 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.517 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:654ddc28:::benchmark_data_smithi039_455948_object147:head# 2024-09-16T17:22:43.517 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.567 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:656a01f0:::benchmark_data_smithi039_455948_object316:head# 2024-09-16T17:22:43.567 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.576 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6667fb4d:::benchmark_data_smithi039_455948_object252:head# 2024-09-16T17:22:43.576 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.584 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:66719837:::benchmark_data_smithi039_455948_object179:head# 2024-09-16T17:22:43.584 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.592 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:66a455f7:::benchmark_data_smithi039_461800_object31:head# 2024-09-16T17:22:43.592 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.601 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:67fc60d1:::benchmark_data_smithi039_455948_object103:head# 2024-09-16T17:22:43.601 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.609 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:67fc91c1:::benchmark_data_smithi039_455948_object52:head# 2024-09-16T17:22:43.609 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.617 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6815ad2e:::benchmark_data_smithi039_455948_object196:head# 2024-09-16T17:22:43.617 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.626 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:69293e2b:::benchmark_data_smithi039_455948_object290:head# 2024-09-16T17:22:43.626 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.634 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6ae4f661:::benchmark_data_smithi039_455948_object241:head# 2024-09-16T17:22:43.634 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.642 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6cb11b42:::benchmark_data_smithi039_455948_object285:head# 2024-09-16T17:22:43.642 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.651 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6ce0dbf0:::benchmark_data_smithi039_455948_object268:head# 2024-09-16T17:22:43.651 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.659 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6d2fe8c5:::benchmark_data_smithi039_455948_object273:head# 2024-09-16T17:22:43.659 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.667 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6d540a4e:::benchmark_data_smithi039_455948_object8:head# 2024-09-16T17:22:43.667 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.676 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6d724163:::benchmark_data_smithi039_455948_object269:head# 2024-09-16T17:22:43.676 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.684 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6e38bb84:::benchmark_data_smithi039_461800_object102:head# 2024-09-16T17:22:43.684 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.692 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6e61b9d5:::benchmark_data_smithi039_455948_object68:head# 2024-09-16T17:22:43.693 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.701 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6f095114:::benchmark_data_smithi039_455948_object94:head# 2024-09-16T17:22:43.701 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.709 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:70691fc8:::benchmark_data_smithi039_455948_object145:head# 2024-09-16T17:22:43.709 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.718 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:708034b3:::benchmark_data_smithi039_455948_object276:head# 2024-09-16T17:22:43.718 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.726 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:70f55f31:::benchmark_data_smithi039_455948_object239:head# 2024-09-16T17:22:43.726 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.734 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:710d1bd1:::benchmark_data_smithi039_461800_object63:head# 2024-09-16T17:22:43.734 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.742 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:713ede28:::benchmark_data_smithi039_461800_object99:head# 2024-09-16T17:22:43.743 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.751 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7149ae21:::benchmark_data_smithi039_461800_object84:head# 2024-09-16T17:22:43.751 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.759 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:718c00a4:::benchmark_data_smithi039_461800_object127:head# 2024-09-16T17:22:43.759 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.768 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:71b1cb82:::benchmark_data_smithi039_455948_object65:head# 2024-09-16T17:22:43.768 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.776 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:71ce8906:::benchmark_data_smithi039_455948_object78:head# 2024-09-16T17:22:43.776 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:72804f95:::benchmark_data_smithi039_461800_object22:head# 2024-09-16T17:22:43.784 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.793 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:72fdddd5:::benchmark_data_smithi039_455948_object86:head# 2024-09-16T17:22:43.793 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.801 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:732a665c:::benchmark_data_smithi039_461800_object26:head# 2024-09-16T17:22:43.801 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.809 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7377856f:::benchmark_data_smithi039_461800_object15:head# 2024-09-16T17:22:43.809 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.818 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:73b633f2:::benchmark_data_smithi039_455948_object150:head# 2024-09-16T17:22:43.818 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.826 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:754b4658:::benchmark_data_smithi039_455948_object77:head# 2024-09-16T17:22:43.826 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.834 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:75a63f9f:::benchmark_data_smithi039_455948_object226:head# 2024-09-16T17:22:43.835 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.843 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:75f9bf67:::benchmark_data_smithi039_455948_object242:head# 2024-09-16T17:22:43.843 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.851 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:76321804:::benchmark_data_smithi039_461800_object49:head# 2024-09-16T17:22:43.851 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.860 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:764ab525:::benchmark_data_smithi039_455948_object317:head# 2024-09-16T17:22:43.860 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.868 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:768e2761:::benchmark_data_smithi039_455948_object238:head# 2024-09-16T17:22:43.868 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.876 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:77182e0f:::benchmark_data_smithi039_455948_object140:head# 2024-09-16T17:22:43.876 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.885 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7739f249:::benchmark_data_smithi039_455948_object34:head# 2024-09-16T17:22:43.885 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.893 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:784592ec:::benchmark_data_smithi039_455948_object33:head# 2024-09-16T17:22:43.893 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.901 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:78c85cc5:::benchmark_data_smithi039_461800_object147:head# 2024-09-16T17:22:43.901 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.927 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7990ab76:::benchmark_data_smithi039_461800_object167:head# 2024-09-16T17:22:43.928 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.943 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7995bd94:::benchmark_data_smithi039_461800_object57:head# 2024-09-16T17:22:43.943 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.951 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7a597a33:::benchmark_data_smithi039_455948_object274:head# 2024-09-16T17:22:43.951 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.960 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7b291d50:::benchmark_data_smithi039_455948_object220:head# 2024-09-16T17:22:43.960 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.968 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7b313b26:::benchmark_data_smithi039_461800_object82:head# 2024-09-16T17:22:43.968 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:43.976 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7bf1daea:::benchmark_data_smithi039_461800_object162:head# 2024-09-16T17:22:43.977 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.018 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7c07aaad:::benchmark_data_smithi039_455948_object91:head# 2024-09-16T17:22:44.018 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.026 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7cd2d272:::benchmark_data_smithi039_455948_object53:head# 2024-09-16T17:22:44.027 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.035 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7d0d8840:::benchmark_data_smithi039_455948_object264:head# 2024-09-16T17:22:44.035 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.043 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7d2f1a11:::benchmark_data_smithi039_461800_object163:head# 2024-09-16T17:22:44.043 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.051 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7d342ae1:::benchmark_data_smithi039_455948_object47:head# 2024-09-16T17:22:44.052 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.060 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7ee172f3:::benchmark_data_smithi039_455948_object304:head# 2024-09-16T17:22:44.060 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.068 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7fe7c24d:::benchmark_data_smithi039_455948_object88:head# 2024-09-16T17:22:44.068 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.077 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:808f8c5a:::benchmark_data_smithi039_461800_object4:head# 2024-09-16T17:22:44.077 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.085 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:80b28505:::benchmark_data_smithi039_461800_object59:head# 2024-09-16T17:22:44.085 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.093 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81444e03:::benchmark_data_smithi039_455948_object191:head# 2024-09-16T17:22:44.093 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.102 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81524550:::benchmark_data_smithi039_455948_object291:head# 2024-09-16T17:22:44.102 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.110 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81688782:::benchmark_data_smithi039_455948_object114:head# 2024-09-16T17:22:44.110 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.118 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81bb4070:::benchmark_data_smithi039_461800_object46:head# 2024-09-16T17:22:44.118 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.127 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:82380bb6:::benchmark_data_smithi039_461800_object12:head# 2024-09-16T17:22:44.127 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.135 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8340282c:::benchmark_data_smithi039_455948_object261:head# 2024-09-16T17:22:44.135 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.143 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:84461d11:::benchmark_data_smithi039_455948_object236:head# 2024-09-16T17:22:44.144 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.152 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:85152b24:::benchmark_data_smithi039_461800_object125:head# 2024-09-16T17:22:44.152 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.202 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:85bbfc69:::benchmark_data_smithi039_455948_object30:head# 2024-09-16T17:22:44.202 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.210 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:861aea5f:::benchmark_data_smithi039_455948_object258:head# 2024-09-16T17:22:44.210 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.219 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8663e6e8:::benchmark_data_smithi039_461800_object112:head# 2024-09-16T17:22:44.219 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.227 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:869e8a5e:::benchmark_data_smithi039_455948_object167:head# 2024-09-16T17:22:44.227 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.235 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:872ef1cc:::benchmark_data_smithi039_455948_object195:head# 2024-09-16T17:22:44.235 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.244 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:87707c04:::benchmark_data_smithi039_461800_object38:head# 2024-09-16T17:22:44.244 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.252 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8923ed3b:::benchmark_data_smithi039_455948_object5:head# 2024-09-16T17:22:44.252 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.260 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:899902b6:::benchmark_data_smithi039_461800_object13:head# 2024-09-16T17:22:44.260 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.269 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:89be3f00:::benchmark_data_smithi039_461800_object2:head# 2024-09-16T17:22:44.269 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.277 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8a3ec687:::benchmark_data_smithi039_455948_object214:head# 2024-09-16T17:22:44.277 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.285 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8aec744d:::benchmark_data_smithi039_455948_object254:head# 2024-09-16T17:22:44.285 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.294 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8c1746e2:::benchmark_data_smithi039_461800_object19:head# 2024-09-16T17:22:44.294 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.302 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8c182888:::benchmark_data_smithi039_461800_object60:head# 2024-09-16T17:22:44.302 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.310 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8c74c0d6:::benchmark_data_smithi039_455948_object116:head# 2024-09-16T17:22:44.311 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.319 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8c7d0d6d:::benchmark_data_smithi039_461800_object35:head# 2024-09-16T17:22:44.319 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.327 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8eb4657a:::benchmark_data_smithi039_455948_object163:head# 2024-09-16T17:22:44.327 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.335 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8ebb2bdf:::benchmark_data_smithi039_461800_object132:head# 2024-09-16T17:22:44.336 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.344 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92b8125e:::benchmark_data_smithi039_455948_object165:head# 2024-09-16T17:22:44.344 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.352 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92dd79ef:::benchmark_data_smithi039_455948_object130:head# 2024-09-16T17:22:44.352 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.361 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92ec70a0:::benchmark_data_smithi039_455948_object170:head# 2024-09-16T17:22:44.361 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.369 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92f0e148:::benchmark_data_smithi039_455948_object223:head# 2024-09-16T17:22:44.369 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.377 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:930eb7ac:::benchmark_data_smithi039_455948_object262:head# 2024-09-16T17:22:44.377 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.386 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:943cbef4:::benchmark_data_smithi039_461800_object66:head# 2024-09-16T17:22:44.386 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.394 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:972245e1:::benchmark_data_smithi039_455948_object7:head# 2024-09-16T17:22:44.394 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.402 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:972cbf49:::benchmark_data_smithi039_455948_object3:head# 2024-09-16T17:22:44.403 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.411 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:97809665:::benchmark_data_smithi039_461800_object176:head# 2024-09-16T17:22:44.411 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.419 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:984c6070:::benchmark_data_smithi039_455948_object1:head# 2024-09-16T17:22:44.419 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.427 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:98946638:::benchmark_data_smithi039_455948_object219:head# 2024-09-16T17:22:44.428 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.444 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:98a59f74:::benchmark_data_smithi039_455948_object216:head# 2024-09-16T17:22:44.444 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.453 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:98b0eb5d:::benchmark_data_smithi039_455948_object131:head# 2024-09-16T17:22:44.453 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.461 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:99654740:::benchmark_data_smithi039_455948_object141:head# 2024-09-16T17:22:44.461 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.469 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9aaac537:::benchmark_data_smithi039_461800_object58:head# 2024-09-16T17:22:44.469 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.478 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9b110e6f:::benchmark_data_smithi039_455948_object55:head# 2024-09-16T17:22:44.478 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.486 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9c86eb42:::benchmark_data_smithi039_455948_object287:head# 2024-09-16T17:22:44.486 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.494 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9c901805:::benchmark_data_smithi039_455948_object59:head# 2024-09-16T17:22:44.494 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.502 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9cc334c3:::benchmark_data_smithi039_461800_object87:head# 2024-09-16T17:22:44.503 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.511 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9d050ed9:::benchmark_data_smithi039_455948_object284:head# 2024-09-16T17:22:44.511 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.519 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9e116042:::benchmark_data_smithi039_455948_object286:head# 2024-09-16T17:22:44.519 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.528 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9eac4879:::benchmark_data_smithi039_455948_object289:head# 2024-09-16T17:22:44.528 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.536 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9f0bc29c:::benchmark_data_smithi039_455948_object257:head# 2024-09-16T17:22:44.536 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.544 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9f254ec7:::benchmark_data_smithi039_461800_object1:head# 2024-09-16T17:22:44.544 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.553 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9fe1d67f:::benchmark_data_smithi039_461800_object0:head# 2024-09-16T17:22:44.553 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.561 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a0acf962:::benchmark_data_smithi039_455948_object61:head# 2024-09-16T17:22:44.561 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.569 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a0b1ff33:::benchmark_data_smithi039_455948_object133:head# 2024-09-16T17:22:44.570 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.578 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a229396d:::benchmark_data_smithi039_461800_object135:head# 2024-09-16T17:22:44.578 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.586 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a299f295:::benchmark_data_smithi039_455948_object176:head# 2024-09-16T17:22:44.586 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.595 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a3610da1:::benchmark_data_smithi039_455948_object149:head# 2024-09-16T17:22:44.595 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.603 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a3616c00:::benchmark_data_smithi039_455948_object246:head# 2024-09-16T17:22:44.603 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.611 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a37a51d4:::benchmark_data_smithi039_461800_object61:head# 2024-09-16T17:22:44.611 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.619 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a3d8a56b:::benchmark_data_smithi039_455948_object73:head# 2024-09-16T17:22:44.620 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.628 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a437720d:::benchmark_data_smithi039_455948_object13:head# 2024-09-16T17:22:44.628 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.636 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a44c4b31:::benchmark_data_smithi039_461800_object14:head# 2024-09-16T17:22:44.636 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.645 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5276dd6:::benchmark_data_smithi039_455948_object2:head# 2024-09-16T17:22:44.645 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.653 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5e30e12:::benchmark_data_smithi039_455948_object17:head# 2024-09-16T17:22:44.653 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.661 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5e964e3:::benchmark_data_smithi039_455948_object45:head# 2024-09-16T17:22:44.661 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.670 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a5f6a828:::benchmark_data_smithi039_455948_object25:head# 2024-09-16T17:22:44.670 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.678 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a63079ff:::benchmark_data_smithi039_455948_object265:head# 2024-09-16T17:22:44.678 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.707 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a64c1b86:::benchmark_data_smithi039_455948_object288:head# 2024-09-16T17:22:44.707 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.749 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a668a0c5:::benchmark_data_smithi039_461800_object20:head# 2024-09-16T17:22:44.749 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.796 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a6698c47:::benchmark_data_smithi039_455948_object259:head# 2024-09-16T17:22:44.796 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.849 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a69d0e24:::benchmark_data_smithi039_455948_object113:head# 2024-09-16T17:22:44.850 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.895 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a6c2f07b:::benchmark_data_smithi039_455948_object212:head# 2024-09-16T17:22:44.895 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:44.941 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a8211e63:::benchmark_data_smithi039_455948_object281:head# 2024-09-16T17:22:44.941 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.011 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a9b29118:::benchmark_data_smithi039_461800_object52:head# 2024-09-16T17:22:45.011 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.084 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a9f1c778:::benchmark_data_smithi039_461800_object107:head# 2024-09-16T17:22:45.084 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.103 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aa4113d5:::benchmark_data_smithi039_461800_object104:head# 2024-09-16T17:22:45.103 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.120 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aba735d4:::benchmark_data_smithi039_461800_object27:head# 2024-09-16T17:22:45.120 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.136 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:abd255dc:::benchmark_data_smithi039_461800_object83:head# 2024-09-16T17:22:45.136 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.153 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ad13fc26:::benchmark_data_smithi039_455948_object110:head# 2024-09-16T17:22:45.153 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.170 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ad8239d1:::benchmark_data_smithi039_455948_object232:head# 2024-09-16T17:22:45.170 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.242 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ad91945e:::benchmark_data_smithi039_455948_object93:head# 2024-09-16T17:22:45.242 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.287 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:adaffd31:::benchmark_data_smithi039_455948_object26:head# 2024-09-16T17:22:45.287 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.328 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ae18ef59:::benchmark_data_smithi039_461800_object17:head# 2024-09-16T17:22:45.328 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.345 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aeb14177:::benchmark_data_smithi039_455948_object29:head# 2024-09-16T17:22:45.345 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.362 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aecd8d89:::benchmark_data_smithi039_455948_object174:head# 2024-09-16T17:22:45.362 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.378 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b03ef840:::benchmark_data_smithi039_455948_object295:head# 2024-09-16T17:22:45.378 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.395 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b089b62d:::benchmark_data_smithi039_461800_object7:head# 2024-09-16T17:22:45.395 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.411 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b0c8cac8:::benchmark_data_smithi039_455948_object117:head# 2024-09-16T17:22:45.412 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.428 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b134289d:::benchmark_data_smithi039_461800_object150:head# 2024-09-16T17:22:45.428 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.482 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b137eb01:::benchmark_data_smithi039_455948_object211:head# 2024-09-16T17:22:45.482 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.520 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b2347b11:::benchmark_data_smithi039_455948_object301:head# 2024-09-16T17:22:45.520 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.568 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b249856d:::benchmark_data_smithi039_455948_object235:head# 2024-09-16T17:22:45.568 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.595 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b2bcdbb8:::benchmark_data_smithi039_461800_object114:head# 2024-09-16T17:22:45.595 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.612 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b3253c9c:::benchmark_data_smithi039_455948_object255:head# 2024-09-16T17:22:45.612 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.628 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b3578fa0:::benchmark_data_smithi039_461800_object121:head# 2024-09-16T17:22:45.628 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.645 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b4899b76:::benchmark_data_smithi039_461800_object165:head# 2024-09-16T17:22:45.645 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.662 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b522d3d2:::benchmark_data_smithi039_461800_object152:head# 2024-09-16T17:22:45.662 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.678 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b5b57515:::benchmark_data_smithi039_455948_object129:head# 2024-09-16T17:22:45.679 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.695 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b5ebc165:::benchmark_data_smithi039_461800_object64:head# 2024-09-16T17:22:45.695 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.712 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b6a40ada:::benchmark_data_smithi039_461800_object76:head# 2024-09-16T17:22:45.712 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.728 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b6c46095:::benchmark_data_smithi039_455948_object178:head# 2024-09-16T17:22:45.729 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.745 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b6ce9049:::benchmark_data_smithi039_455948_object172:head# 2024-09-16T17:22:45.745 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.762 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b6cf5c8c:::benchmark_data_smithi039_455948_object231:head# 2024-09-16T17:22:45.762 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.779 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b7c8f422:::benchmark_data_smithi039_455948_object112:head# 2024-09-16T17:22:45.779 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.795 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b811e841:::benchmark_data_smithi039_461800_object134:head# 2024-09-16T17:22:45.795 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.812 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-16T17:22:45.812 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.829 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b8b98acb:::benchmark_data_smithi039_455948_object313:head# 2024-09-16T17:22:45.829 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.845 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b9991d10:::benchmark_data_smithi039_455948_object154:head# 2024-09-16T17:22:45.845 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.862 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b9d0eac5:::benchmark_data_smithi039_461800_object145:head# 2024-09-16T17:22:45.862 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.879 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b9f36334:::benchmark_data_smithi039_461800_object91:head# 2024-09-16T17:22:45.879 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.895 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bad4f01e:::benchmark_data_smithi039_461800_object29:head# 2024-09-16T17:22:45.895 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.912 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:baf26fd7:::benchmark_data_smithi039_455948_object134:head# 2024-09-16T17:22:45.912 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.929 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bb71b590:::benchmark_data_smithi039_455948_object314:head# 2024-09-16T17:22:45.929 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.945 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bba0e1da:::benchmark_data_smithi039_461800_object6:head# 2024-09-16T17:22:45.945 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.962 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bba586dd:::benchmark_data_smithi039_461800_object81:head# 2024-09-16T17:22:45.962 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.979 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bc052407:::benchmark_data_smithi039_455948_object270:head# 2024-09-16T17:22:45.979 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:45.995 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bcff6325:::benchmark_data_smithi039_455948_object315:head# 2024-09-16T17:22:45.995 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.012 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bd19ab08:::benchmark_data_smithi039_455948_object81:head# 2024-09-16T17:22:46.012 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.029 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:be7390f5:::benchmark_data_smithi039_455948_object83:head# 2024-09-16T17:22:46.029 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.045 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bf8a7830:::benchmark_data_smithi039_461800_object41:head# 2024-09-16T17:22:46.045 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.062 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c07324f3:::benchmark_data_smithi039_455948_object302:head# 2024-09-16T17:22:46.062 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.079 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c094cc24:::benchmark_data_smithi039_461800_object89:head# 2024-09-16T17:22:46.079 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.096 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c0d972fb:::benchmark_data_smithi039_455948_object209:head# 2024-09-16T17:22:46.096 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.112 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c1b3a68a:::benchmark_data_smithi039_461800_object160:head# 2024-09-16T17:22:46.112 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.129 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c34a47c1:::benchmark_data_smithi039_455948_object319:head# 2024-09-16T17:22:46.129 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.146 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c38689dc:::benchmark_data_smithi039_455948_object50:head# 2024-09-16T17:22:46.146 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.162 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c3893c52:::benchmark_data_smithi039_455948_object66:head# 2024-09-16T17:22:46.162 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.179 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c464c3ce:::benchmark_data_smithi039_455948_object106:head# 2024-09-16T17:22:46.179 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.196 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c496e5b5:::benchmark_data_smithi039_461800_object110:head# 2024-09-16T17:22:46.196 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.212 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c4a7d2e8:::benchmark_data_smithi039_461800_object96:head# 2024-09-16T17:22:46.212 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.229 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c56ee08e:::benchmark_data_smithi039_455948_object125:head# 2024-09-16T17:22:46.229 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.254 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c6555459:::benchmark_data_smithi039_455948_object282:head# 2024-09-16T17:22:46.254 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.271 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c6ec286a:::benchmark_data_smithi039_461800_object174:head# 2024-09-16T17:22:46.271 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.287 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c70cd8e6:::benchmark_data_smithi039_455948_object181:head# 2024-09-16T17:22:46.288 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.304 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c72b9798:::benchmark_data_smithi039_455948_object186:head# 2024-09-16T17:22:46.304 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.321 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c7547bd4:::benchmark_data_smithi039_455948_object160:head# 2024-09-16T17:22:46.321 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.338 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c8cc5007:::benchmark_data_smithi039_461800_object128:head# 2024-09-16T17:22:46.338 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.354 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c8d50a66:::benchmark_data_smithi039_455948_object15:head# 2024-09-16T17:22:46.354 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.397 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c916bf36:::benchmark_data_smithi039_455948_object300:head# 2024-09-16T17:22:46.397 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.413 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c98347ce:::benchmark_data_smithi039_455948_object44:head# 2024-09-16T17:22:46.413 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.429 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ca283ebf:::benchmark_data_smithi039_455948_object245:head# 2024-09-16T17:22:46.430 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.446 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cb190b2d:::benchmark_data_smithi039_455948_object299:head# 2024-09-16T17:22:46.446 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.463 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cb44df0b:::benchmark_data_smithi039_455948_object102:head# 2024-09-16T17:22:46.463 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.479 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cb8dc5ee:::benchmark_data_smithi039_455948_object201:head# 2024-09-16T17:22:46.479 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.496 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cd24ea32:::benchmark_data_smithi039_455948_object42:head# 2024-09-16T17:22:46.496 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.513 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cd94f270:::benchmark_data_smithi039_455948_object124:head# 2024-09-16T17:22:46.513 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.529 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cdc23a9a:::benchmark_data_smithi039_461800_object32:head# 2024-09-16T17:22:46.530 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.546 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ce974f23:::benchmark_data_smithi039_455948_object159:head# 2024-09-16T17:22:46.546 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.563 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cebe2675:::benchmark_data_smithi039_461800_object140:head# 2024-09-16T17:22:46.563 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.579 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cf1dabff:::benchmark_data_smithi039_455948_object151:head# 2024-09-16T17:22:46.580 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.596 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cf56f22e:::benchmark_data_smithi039_455948_object294:head# 2024-09-16T17:22:46.596 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.613 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cfee26d6:::benchmark_data_smithi039_461800_object95:head# 2024-09-16T17:22:46.613 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.630 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d04eceaf:::benchmark_data_smithi039_455948_object39:head# 2024-09-16T17:22:46.630 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.646 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d06d3616:::benchmark_data_smithi039_455948_object24:head# 2024-09-16T17:22:46.646 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.663 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d08457ea:::benchmark_data_smithi039_455948_object4:head# 2024-09-16T17:22:46.663 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.680 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d0851a13:::benchmark_data_smithi039_455948_object306:head# 2024-09-16T17:22:46.680 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.696 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d08dfc7d:::benchmark_data_smithi039_455948_object89:head# 2024-09-16T17:22:46.696 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.713 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d0bdfad3:::benchmark_data_smithi039_461800_object109:head# 2024-09-16T17:22:46.713 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.729 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d1c0a25d:::benchmark_data_smithi039_455948_object105:head# 2024-09-16T17:22:46.730 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.746 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d2347595:::benchmark_data_smithi039_455948_object123:head# 2024-09-16T17:22:46.746 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.763 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d26aabdc:::benchmark_data_smithi039_461800_object168:head# 2024-09-16T17:22:46.763 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.780 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d2987685:::benchmark_data_smithi039_461800_object172:head# 2024-09-16T17:22:46.780 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.796 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d2a4d7e3:::benchmark_data_smithi039_461800_object106:head# 2024-09-16T17:22:46.796 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.813 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d2f20841:::benchmark_data_smithi039_455948_object95:head# 2024-09-16T17:22:46.813 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.830 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d33b721d:::benchmark_data_smithi039_455948_object244:head# 2024-09-16T17:22:46.830 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.846 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d37b01a7:::benchmark_data_smithi039_455948_object243:head# 2024-09-16T17:22:46.847 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.863 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d432d0ae:::benchmark_data_smithi039_455948_object99:head# 2024-09-16T17:22:46.863 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.880 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d4cfa604:::benchmark_data_smithi039_455948_object14:head# 2024-09-16T17:22:46.880 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.896 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d55839d8:::benchmark_data_smithi039_461800_object129:head# 2024-09-16T17:22:46.896 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.913 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d66a0831:::benchmark_data_smithi039_455948_object143:head# 2024-09-16T17:22:46.913 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.930 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d7aed95b:::benchmark_data_smithi039_461800_object71:head# 2024-09-16T17:22:46.930 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.946 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d7dff5fe:::benchmark_data_smithi039_455948_object51:head# 2024-09-16T17:22:46.946 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.963 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d81d2707:::benchmark_data_smithi039_455948_object248:head# 2024-09-16T17:22:46.963 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.980 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d8cdeb85:::benchmark_data_smithi039_455948_object279:head# 2024-09-16T17:22:46.980 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:46.996 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d95fa43b:::benchmark_data_smithi039_455948_object97:head# 2024-09-16T17:22:46.997 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.013 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:da4991aa:::benchmark_data_smithi039_461800_object54:head# 2024-09-16T17:22:47.013 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.030 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:db0a03c3:::benchmark_data_smithi039_455948_object35:head# 2024-09-16T17:22:47.030 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.046 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:db111d65:::benchmark_data_smithi039_461800_object86:head# 2024-09-16T17:22:47.047 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.063 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:db62d852:::benchmark_data_smithi039_461800_object159:head# 2024-09-16T17:22:47.063 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.080 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dbe2ac53:::benchmark_data_smithi039_455948_object92:head# 2024-09-16T17:22:47.080 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.096 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dc7e8981:::benchmark_data_smithi039_461800_object53:head# 2024-09-16T17:22:47.097 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.113 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dcfb2107:::benchmark_data_smithi039_461800_object118:head# 2024-09-16T17:22:47.113 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.130 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:de97e1fb:::benchmark_data_smithi039_461800_object3:head# 2024-09-16T17:22:47.130 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.147 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e003ea5b:::benchmark_data_smithi039_461800_object11:head# 2024-09-16T17:22:47.147 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.163 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e038408c:::benchmark_data_smithi039_461800_object40:head# 2024-09-16T17:22:47.163 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.180 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e05e2d0e:::benchmark_data_smithi039_455948_object127:head# 2024-09-16T17:22:47.180 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.230 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e0c79d68:::benchmark_data_smithi039_461800_object93:head# 2024-09-16T17:22:47.230 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.272 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e0f1cb6f:::benchmark_data_smithi039_461800_object169:head# 2024-09-16T17:22:47.272 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.288 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e18f9e35:::benchmark_data_smithi039_455948_object82:head# 2024-09-16T17:22:47.289 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.305 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e1a4e1b7:::benchmark_data_smithi039_455948_object189:head# 2024-09-16T17:22:47.305 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.322 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e2081651:::benchmark_data_smithi039_455948_object303:head# 2024-09-16T17:22:47.322 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.339 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e22d9d39:::benchmark_data_smithi039_461800_object44:head# 2024-09-16T17:22:47.339 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.355 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e3634a32:::benchmark_data_smithi039_461800_object48:head# 2024-09-16T17:22:47.355 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.372 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e36479d3:::benchmark_data_smithi039_455948_object277:head# 2024-09-16T17:22:47.372 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.426 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e4509956:::benchmark_data_smithi039_455948_object222:head# 2024-09-16T17:22:47.427 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.472 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e45ad9a8:::benchmark_data_smithi039_455948_object310:head# 2024-09-16T17:22:47.472 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.527 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e49f1f25:::benchmark_data_smithi039_455948_object60:head# 2024-09-16T17:22:47.527 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.547 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e65ee987:::benchmark_data_smithi039_461800_object5:head# 2024-09-16T17:22:47.547 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.564 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e7a4dce5:::benchmark_data_smithi039_461800_object34:head# 2024-09-16T17:22:47.564 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.580 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e8279365:::benchmark_data_smithi039_455948_object162:head# 2024-09-16T17:22:47.580 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.597 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e84b1af3:::benchmark_data_smithi039_455948_object168:head# 2024-09-16T17:22:47.597 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.614 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e860450e:::benchmark_data_smithi039_455948_object40:head# 2024-09-16T17:22:47.614 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.639 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e9546805:::benchmark_data_smithi039_455948_object84:head# 2024-09-16T17:22:47.639 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.655 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e9c31f48:::benchmark_data_smithi039_455948_object57:head# 2024-09-16T17:22:47.655 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.672 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e9db3ae8:::benchmark_data_smithi039_455948_object185:head# 2024-09-16T17:22:47.673 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.689 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ea00b668:::benchmark_data_smithi039_455948_object126:head# 2024-09-16T17:22:47.689 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.705 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ea038be8:::benchmark_data_smithi039_461800_object103:head# 2024-09-16T17:22:47.706 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.722 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ea427d4a:::benchmark_data_smithi039_461800_object25:head# 2024-09-16T17:22:47.722 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.739 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:eac552d0:::benchmark_data_smithi039_455948_object217:head# 2024-09-16T17:22:47.739 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.756 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ed73d1bd:::benchmark_data_smithi039_461800_object117:head# 2024-09-16T17:22:47.756 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.772 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ed7b1e48:::benchmark_data_smithi039_461800_object157:head# 2024-09-16T17:22:47.772 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.789 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ed7c17e0:::benchmark_data_smithi039_455948_object152:head# 2024-09-16T17:22:47.789 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.805 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:edc14e53:::benchmark_data_smithi039_455948_object230:head# 2024-09-16T17:22:47.806 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.822 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ee810f8e:::benchmark_data_smithi039_461800_object173:head# 2024-09-16T17:22:47.822 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.839 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ee9608a4:::benchmark_data_smithi039_461800_object65:head# 2024-09-16T17:22:47.839 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.856 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:eebb0eed:::benchmark_data_smithi039_461800_object137:head# 2024-09-16T17:22:47.856 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.872 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:efd04b1c:::benchmark_data_smithi039_455948_object225:head# 2024-09-16T17:22:47.872 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.889 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:efea2d35:::benchmark_data_smithi039_455948_object146:head# 2024-09-16T17:22:47.889 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.906 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:eff1383c:::benchmark_data_smithi039_461800_object166:head# 2024-09-16T17:22:47.906 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.923 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f01ad973:::benchmark_data_smithi039_461800_object131:head# 2024-09-16T17:22:47.923 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.939 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f1bb07dd:::benchmark_data_smithi039_455948_object298:head# 2024-09-16T17:22:47.939 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.956 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f1e99f50:::benchmark_data_smithi039_455948_object215:head# 2024-09-16T17:22:47.956 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.972 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f1f5e565:::benchmark_data_smithi039_455948_object307:head# 2024-09-16T17:22:47.973 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:47.989 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f20c71ff:::benchmark_data_smithi039_455948_object267:head# 2024-09-16T17:22:47.989 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.006 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f274efb2:::benchmark_data_smithi039_455948_object27:head# 2024-09-16T17:22:48.006 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.023 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f2a8576b:::benchmark_data_smithi039_455948_object54:head# 2024-09-16T17:22:48.023 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.039 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f2f0a09c:::benchmark_data_smithi039_455948_object227:head# 2024-09-16T17:22:48.039 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.056 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f39faa1c:::benchmark_data_smithi039_461800_object151:head# 2024-09-16T17:22:48.056 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.072 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f401f323:::benchmark_data_smithi039_461800_object36:head# 2024-09-16T17:22:48.073 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.089 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f5996306:::benchmark_data_smithi039_455948_object22:head# 2024-09-16T17:22:48.089 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.106 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f5dc5fcb:::benchmark_data_smithi039_461800_object156:head# 2024-09-16T17:22:48.106 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.122 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f5ffe507:::benchmark_data_smithi039_461800_object116:head# 2024-09-16T17:22:48.123 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.139 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f61ef65d:::benchmark_data_smithi039_455948_object111:head# 2024-09-16T17:22:48.139 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.156 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f621bc7f:::benchmark_data_smithi039_455948_object182:head# 2024-09-16T17:22:48.156 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.172 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f6edecad:::benchmark_data_smithi039_461800_object111:head# 2024-09-16T17:22:48.173 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.189 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f7044b00:::benchmark_data_smithi039_461800_object138:head# 2024-09-16T17:22:48.189 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.206 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f72c3135:::benchmark_data_smithi039_455948_object21:head# 2024-09-16T17:22:48.206 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.222 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f7b30eee:::benchmark_data_smithi039_455948_object194:head# 2024-09-16T17:22:48.223 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.239 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f97e448b:::benchmark_data_smithi039_455948_object10:head# 2024-09-16T17:22:48.239 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.256 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f98eff84:::benchmark_data_smithi039_461800_object16:head# 2024-09-16T17:22:48.272 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.273 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f9d8cd97:::benchmark_data_smithi039_455948_object199:head# 2024-09-16T17:22:48.273 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.289 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f9e3593c:::benchmark_data_smithi039_455948_object256:head# 2024-09-16T17:22:48.289 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.306 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fa9bf67e:::benchmark_data_smithi039_455948_object32:head# 2024-09-16T17:22:48.306 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.323 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fadac265:::benchmark_data_smithi039_455948_object305:head# 2024-09-16T17:22:48.323 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.339 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fadc5b0b:::benchmark_data_smithi039_455948_object104:head# 2024-09-16T17:22:48.339 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.389 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fc574797:::benchmark_data_smithi039_461800_object62:head# 2024-09-16T17:22:48.390 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.406 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fc984da2:::benchmark_data_smithi039_461800_object37:head# 2024-09-16T17:22:48.406 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.423 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fd05df8b:::benchmark_data_smithi039_455948_object70:head# 2024-09-16T17:22:48.423 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.439 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fdea465d:::benchmark_data_smithi039_455948_object107:head# 2024-09-16T17:22:48.440 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.456 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fe001570:::benchmark_data_smithi039_455948_object171:head# 2024-09-16T17:22:48.456 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.473 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ff9caaec:::benchmark_data_smithi039_455948_object263:head# 2024-09-16T17:22:48.473 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:22:48.490 INFO:tasks.workunit.client.0.smithi039.stdout:write_pg epoch 224 info 1.0( v 224'339 (9'161,224'339] local-lis/les=23/24 n=498 ec=7/7 lis/c=23/23 les/c/f=24/24/0 sis=23) 2024-09-16T17:22:48.515 INFO:tasks.workunit.client.0.smithi039.stdout:Import successful 2024-09-16T17:22:49.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:157: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-09-16T17:22:49.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-16T17:22:49.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T17:22:49.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T17:22:49.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T17:22:49.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-16T17:22:49.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:22:49.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:22:49.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:22:49.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:22:49.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-16T17:22:49.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-16T17:22:49.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T17:22:49.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T17:22:49.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-16T17:22:49.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T17:22:49.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:22:49.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:22:49.058 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:22:49.058 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:22:49.058 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:22:49.058 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:22:49.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:22:49.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:22:49.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-16T17:22:49.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-16T17:22:49.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:22:49.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:22:49.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:22:49.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:22:49.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T17:22:49.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T17:22:49.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-16T17:22:49.059 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:22:49.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T17:22:49.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:22:49.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-16T17:22:49.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T17:22:49.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T17:22:49.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T17:22:49.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T17:22:49.096 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:49.092+0000 7ff8348e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:22:49.099 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:49.096+0000 7ff8348e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:22:49.101 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:49.096+0000 7ff8348e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:22:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T17:22:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:22:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:22:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:22:49.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:22:49.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:22:49.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:22:49.547 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:22:49.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:22:49.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:22:49.683 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:49.680+0000 7ff8348e07c0 -1 Falling back to public interface 2024-09-16T17:22:49.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:22:50.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:22:50.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:22:50.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:22:50.972 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:22:50.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:22:50.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:22:51.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:22:52.231 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:52.228+0000 7ff8348e07c0 -1 osd.0 228 log_to_monitors true 2024-09-16T17:22:52.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:22:52.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:22:52.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:22:52.398 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:22:52.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:22:52.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:22:52.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:22:53.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:22:53.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:22:53.834 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:22:53.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:22:53.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:22:53.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:22:54.255 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 231 up_thru 231 down_at 229 last_clean_interval [23,228) [v2:127.0.0.1:6802/2299439665,v1:127.0.0.1:6803/2299439665] [v2:127.0.0.1:6804/2299439665,v1:127.0.0.1:6805/2299439665] exists,up 68294d40-1585-4437-81ff-1190991a9a32 2024-09-16T17:22:54.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:22:54.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:22:54.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:22:54.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:159: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-16T17:22:54.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:22:54.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:22:54.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:22:54.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:22:54.259 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:22:54.259 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:22:54.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:22:54.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:22:54.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:22:54.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:22:54.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:22:54.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:22:54.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:22:54.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:22:54.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:22:54.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T17:22:54.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:22:54.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:22:54.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:22:54.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=992137445379 2024-09-16T17:22:54.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 992137445379 2024-09-16T17:22:54.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-992137445379' 2024-09-16T17:22:54.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:22:54.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:22:54.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-992137445379 2024-09-16T17:22:54.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:22:54.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-992137445379 2024-09-16T17:22:54.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:22:54.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=992137445379 2024-09-16T17:22:54.966 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 992137445379 2024-09-16T17:22:54.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 992137445379' 2024-09-16T17:22:54.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:22:55.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 992137445379 -lt 992137445379 2024-09-16T17:22:55.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:22:55.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:22:55.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:22:56.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T17:22:56.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:22:56.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:22:56.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:22:56.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:22:56.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:22:56.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:22:56.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:22:56.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T17:22:56.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:22:56.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:22:56.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:22:57.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T17:22:57.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:22:57.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:22:57.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-09-16T17:22:57.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-16T17:22:57.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:22:57.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-16T17:22:57.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:22:57.029 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:22:57.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:22:57.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:22:57.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:22:57.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:22:57.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:22:57.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:22:57.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:22:57.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:22:57.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:22:57.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:22:57.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:22:57.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:22:57.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:22:57.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:22:57.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:22:57.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:22:57.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-16T17:22:57.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:22:57.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:22:57.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:22:57.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.444977 2024-09-16T17:22:57.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:22:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:22:57.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:22:57.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:22:57.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:22:57.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:22:57.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:22:57.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:22:57.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:22:57.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:22:57.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:22:57.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:22:57.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:22:57.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:22:57.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:22:57.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-16T17:22:57.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:22:57.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:22:57.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:22:57.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.444977 2024-09-16T17:22:57.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:22:57.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:22:57.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-09-16T17:22:57.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:22:57.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:22:57.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:22:57.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.444977 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_split td/pg-split-merge 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:163: TEST_import_after_split: local dir=td/pg-split-merge 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:165: TEST_import_after_split: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-09-16T17:22:57.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T17:22:57.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:22:57.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:22:57.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:22:57.785 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:22:57.785 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:22:57.785 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:22:57.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:22:57.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T17:22:58.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:22:58.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:22:58.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:22:58.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:22:58.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:22:58.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:22:58.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:22:58.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:22:58.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:22:58.032 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:22:58.032 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:22:58.032 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:22:58.032 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.444977/ceph-mon.a.asok 2024-09-16T17:22:58.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:22:58.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.444977/ceph-mon.a.asok config get fsid 2024-09-16T17:22:58.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:22:58.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:22:58.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:22:58.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:22:58.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:22:58.152 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:22:58.152 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:22:58.152 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:22:58.152 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:22:58.152 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:22:58.152 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:22:58.153 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.444977/ceph-mon.a.asok 2024-09-16T17:22:58.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:22:58.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.444977/ceph-mon.a.asok config get mon_host 2024-09-16T17:22:58.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:166: TEST_import_after_split: run_mgr td/pg-split-merge x 2024-09-16T17:22:58.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-09-16T17:22:58.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:22:58.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:22:58.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:22:58.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-09-16T17:22:58.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:22:59.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:22:59.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:22:59.186 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:22:59.186 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:22:59.187 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:22:59.187 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:22:59.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:22:59.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:22:59.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:22:59.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:167: TEST_import_after_split: run_osd td/pg-split-merge 0 2024-09-16T17:22:59.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-16T17:22:59.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:22:59.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:22:59.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:22:59.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-09-16T17:22:59.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:22:59.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:22:59.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:22:59.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:22:59.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-16T17:22:59.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-16T17:22:59.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:22:59.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:22:59.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-16T17:22:59.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:22:59.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:22:59.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:22:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:22:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:22:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:22:59.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:22:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:22:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:22:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:22:59.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:22:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-16T17:22:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-16T17:22:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:22:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:22:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:22:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:22:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:22:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:22:59.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-09-16T17:22:59.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:22:59.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c1945d7e-0f4d-49d8-91bb-17a1555e217a 2024-09-16T17:22:59.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c1945d7e-0f4d-49d8-91bb-17a1555e217a' 2024-09-16T17:22:59.261 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 c1945d7e-0f4d-49d8-91bb-17a1555e217a 2024-09-16T17:22:59.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:22:59.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzaehmfzl/EBAA4El6Caq9ot2YRPQInyq3bg== 2024-09-16T17:22:59.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzaehmfzl/EBAA4El6Caq9ot2YRPQInyq3bg=="}' 2024-09-16T17:22:59.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c1945d7e-0f4d-49d8-91bb-17a1555e217a -i td/pg-split-merge/0/new.json 2024-09-16T17:22:59.617 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:22:59.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-09-16T17:22:59.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzaehmfzl/EBAA4El6Caq9ot2YRPQInyq3bg== --osd-uuid c1945d7e-0f4d-49d8-91bb-17a1555e217a 2024-09-16T17:22:59.666 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:59.660+0000 7fcf36bfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:22:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:59.672+0000 7fcf36bfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:22:59.678 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:59.672+0000 7fcf36bfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:22:59.678 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:59.672+0000 7fcf36bfd7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-16T17:22:59.678 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:59.672+0000 7fcf36bfd7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-16T17:22:59.678 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:59.672+0000 7fcf36bfd7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-16T17:22:59.716 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:22:59.712+0000 7fcf36bfd7c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-09-16T17:23:05.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-09-16T17:23:05.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:23:05.855 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:23:05.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:23:05.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:23:06.450 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:23:06.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:23:06.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:23:06.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:23:06.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:23:06.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:23:06.485 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:23:06.480+0000 7ff32826f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:23:06.487 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:23:06.484+0000 7ff32826f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:23:06.488 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:23:06.484+0000 7ff32826f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:23:06.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:23:06.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:23:06.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:23:06.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:23:06.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:23:06.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:06.939 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:23:06.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:23:06.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:06.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:07.063 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:23:07.060+0000 7ff32826f7c0 -1 Falling back to public interface 2024-09-16T17:23:07.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:08.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:08.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:08.424 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:23:08.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:23:08.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:08.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:08.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:09.699 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:23:09.695+0000 7ff32826f7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:23:09.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:09.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:09.860 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:23:09.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:23:09.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:09.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:10.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:11.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:11.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:11.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:23:11.431 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:23:11.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:11.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:11.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:12.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:12.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:12.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:23:12.894 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:23:12.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:12.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:13.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:14.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:14.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:14.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:23:14.443 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:23:14.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:14.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:14.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:15.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:15.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:15.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:23:15.871 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:23:15.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:15.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:16.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:17.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:17.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:17.449 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:23:17.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:23:17.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:17.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:17.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:18.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:18.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:18.883 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:23:18.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:23:18.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:18.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:19.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:20.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:20.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:20.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:23:20.445 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:23:20.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:20.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:20.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:21.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:21.885 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:23:21.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:21.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:23:21.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:21.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:22.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:23.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:23.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:23.481 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:23:23.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:23:23.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:23.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:24.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:25.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:25.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:25.304 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:23:25.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:23:25.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:25.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:26.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:27.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:27.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:27.317 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:23:27.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:23:27.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:27.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:28.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:29.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:29.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:29.194 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:23:29.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:23:29.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:29.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:30.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:31.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:31.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:31.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:23:31.032 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:23:31.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:31.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:31.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:32.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:32.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:32.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:23:32.717 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:23:32.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:32.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:33.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:34.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:34.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:34.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:23:34.431 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:23:34.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:34.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:34.901 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3647817931,v1:127.0.0.1:6803/3647817931] [v2:127.0.0.1:6804/3647817931,v1:127.0.0.1:6805/3647817931] exists,up c1945d7e-0f4d-49d8-91bb-17a1555e217a 2024-09-16T17:23:34.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:23:34.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:23:34.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:23:34.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:169: TEST_import_after_split: ceph osd pool create foo 1 2024-09-16T17:23:35.488 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-16T17:23:35.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:170: TEST_import_after_split: wait_for_clean 2024-09-16T17:23:35.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:23:35.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:23:35.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:23:35.507 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:23:35.508 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:23:35.508 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:23:35.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:23:35.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:23:35.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:23:35.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:23:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:23:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:23:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:23:35.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:23:35.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:23:36.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T17:23:36.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:23:36.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:23:36.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:23:36.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836487 2024-09-16T17:23:36.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836487 2024-09-16T17:23:36.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836487' 2024-09-16T17:23:36.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:23:36.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836487 2024-09-16T17:23:36.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:23:36.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:23:36.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836487 2024-09-16T17:23:36.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:23:36.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836487 2024-09-16T17:23:36.260 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836487 2024-09-16T17:23:36.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-16T17:23:36.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:23:36.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-16T17:23:36.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:23:37.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:23:37.743 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:23:38.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-16T17:23:38.189 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:23:38.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:23:38.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:23:38.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T17:23:38.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:23:38.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:23:38.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:23:38.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:23:38.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:23:38.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:23:38.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:23:39.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T17:23:39.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:23:39.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:23:39.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:23:39.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T17:23:39.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:23:39.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:23:39.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:171: TEST_import_after_split: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-09-16T17:23:39.779 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-16T17:23:39.779 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-09-16T17:23:39.780 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_468595 2024-09-16T17:23:39.780 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T17:23:39.780 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-16T17:23:40.779 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 112 96 0.0937386 0.09375 0.117412 0.158482 2024-09-16T17:23:41.780 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 225 209 0.102036 0.110352 0.134134 0.143907 2024-09-16T17:23:42.780 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 337 321 0.104476 0.109375 0.117288 0.145009 2024-09-16T17:23:43.780 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 3.08921 2024-09-16T17:23:43.780 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 337 2024-09-16T17:23:43.780 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 1024 2024-09-16T17:23:43.780 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 1024 2024-09-16T17:23:43.780 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.106533 2024-09-16T17:23:43.780 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.00931581 2024-09-16T17:23:43.780 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.110352 2024-09-16T17:23:43.780 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.09375 2024-09-16T17:23:43.780 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 109 2024-09-16T17:23:43.780 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 9.53939 2024-09-16T17:23:43.781 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 113 2024-09-16T17:23:43.781 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 96 2024-09-16T17:23:43.781 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.143695 2024-09-16T17:23:43.781 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.0283447 2024-09-16T17:23:43.781 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 0.2262 2024-09-16T17:23:43.781 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.107248 2024-09-16T17:23:43.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:173: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-09-16T17:23:43.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:23:43.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:23:43.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:23:43.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:23:43.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:23:44.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:23:44.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:174: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-09-16T17:23:45.500 INFO:tasks.workunit.client.0.smithi039.stderr:Exporting 1.0 info 1.0( v 9'338 (0'0,9'338] local-lis/les=7/8 n=338 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-16T17:23:45.506 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:000564e5:::benchmark_data_smithi039_468595_object239:head# 2024-09-16T17:23:45.507 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:006f988d:::benchmark_data_smithi039_468595_object234:head# 2024-09-16T17:23:45.507 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:00e12f2f:::benchmark_data_smithi039_468595_object131:head# 2024-09-16T17:23:45.508 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:028910e9:::benchmark_data_smithi039_468595_object24:head# 2024-09-16T17:23:45.509 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:056c3ba9:::benchmark_data_smithi039_468595_object172:head# 2024-09-16T17:23:45.509 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:05c59347:::benchmark_data_smithi039_468595_object2:head# 2024-09-16T17:23:45.510 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:06eba0d7:::benchmark_data_smithi039_468595_object45:head# 2024-09-16T17:23:45.510 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0751acb2:::benchmark_data_smithi039_468595_object183:head# 2024-09-16T17:23:45.511 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:07c393fb:::benchmark_data_smithi039_468595_object136:head# 2024-09-16T17:23:45.511 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0a3ad74d:::benchmark_data_smithi039_468595_object232:head# 2024-09-16T17:23:45.512 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0a7cf200:::benchmark_data_smithi039_468595_object214:head# 2024-09-16T17:23:45.513 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0bd67d89:::benchmark_data_smithi039_468595_object247:head# 2024-09-16T17:23:45.513 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0c3f0015:::benchmark_data_smithi039_468595_object96:head# 2024-09-16T17:23:45.513 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0c4ca308:::benchmark_data_smithi039_468595_object122:head# 2024-09-16T17:23:45.514 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0c96026c:::benchmark_data_smithi039_468595_object102:head# 2024-09-16T17:23:45.515 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0d76d630:::benchmark_data_smithi039_468595_object34:head# 2024-09-16T17:23:45.515 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0e30d34f:::benchmark_data_smithi039_468595_object218:head# 2024-09-16T17:23:45.516 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:11a6ffc3:::benchmark_data_smithi039_468595_object70:head# 2024-09-16T17:23:45.516 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:13470305:::benchmark_data_smithi039_468595_object140:head# 2024-09-16T17:23:45.517 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:143bf9f0:::benchmark_data_smithi039_468595_object9:head# 2024-09-16T17:23:45.517 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:147e98f9:::benchmark_data_smithi039_468595_object51:head# 2024-09-16T17:23:45.518 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:14ee031d:::benchmark_data_smithi039_468595_object325:head# 2024-09-16T17:23:45.518 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1569fe73:::benchmark_data_smithi039_468595_object101:head# 2024-09-16T17:23:45.519 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:157740f9:::benchmark_data_smithi039_468595_object292:head# 2024-09-16T17:23:45.519 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:15c58c9e:::benchmark_data_smithi039_468595_object323:head# 2024-09-16T17:23:45.519 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:15cba213:::benchmark_data_smithi039_468595_object75:head# 2024-09-16T17:23:45.520 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1682914d:::benchmark_data_smithi039_468595_object1:head# 2024-09-16T17:23:45.520 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:16ca303c:::benchmark_data_smithi039_468595_object236:head# 2024-09-16T17:23:45.521 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:170afdae:::benchmark_data_smithi039_468595_object116:head# 2024-09-16T17:23:45.521 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:18f19464:::benchmark_data_smithi039_468595_object258:head# 2024-09-16T17:23:45.522 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1a549628:::benchmark_data_smithi039_468595_object54:head# 2024-09-16T17:23:45.522 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1b3855f6:::benchmark_data_smithi039_468595_object256:head# 2024-09-16T17:23:45.522 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1b555b63:::benchmark_data_smithi039_468595_object178:head# 2024-09-16T17:23:45.523 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1cbd37a8:::benchmark_data_smithi039_468595_object269:head# 2024-09-16T17:23:45.523 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1cc2b390:::benchmark_data_smithi039_468595_object295:head# 2024-09-16T17:23:45.524 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1dd3e2de:::benchmark_data_smithi039_468595_object108:head# 2024-09-16T17:23:45.524 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:20866bca:::benchmark_data_smithi039_468595_object6:head# 2024-09-16T17:23:45.525 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:20e66408:::benchmark_data_smithi039_468595_object74:head# 2024-09-16T17:23:45.525 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2223a6bd:::benchmark_data_smithi039_468595_object7:head# 2024-09-16T17:23:45.526 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:22b0abf3:::benchmark_data_smithi039_468595_object229:head# 2024-09-16T17:23:45.526 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:22d6afb2:::benchmark_data_smithi039_468595_object243:head# 2024-09-16T17:23:45.527 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2474db63:::benchmark_data_smithi039_468595_object61:head# 2024-09-16T17:23:45.527 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:267ac8e8:::benchmark_data_smithi039_468595_object280:head# 2024-09-16T17:23:45.528 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2716b92a:::benchmark_data_smithi039_468595_object196:head# 2024-09-16T17:23:45.528 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:275da9db:::benchmark_data_smithi039_468595_object261:head# 2024-09-16T17:23:45.529 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2792d397:::benchmark_data_smithi039_468595_object286:head# 2024-09-16T17:23:45.529 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:28dc7dc3:::benchmark_data_smithi039_468595_object204:head# 2024-09-16T17:23:45.530 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2b30d01d:::benchmark_data_smithi039_468595_object170:head# 2024-09-16T17:23:45.530 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2b359d12:::benchmark_data_smithi039_468595_object109:head# 2024-09-16T17:23:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2bc2d150:::benchmark_data_smithi039_468595_object263:head# 2024-09-16T17:23:45.531 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2c298f0c:::benchmark_data_smithi039_468595_object222:head# 2024-09-16T17:23:45.532 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2dfe2ce0:::benchmark_data_smithi039_468595_object180:head# 2024-09-16T17:23:45.532 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2f3a15aa:::benchmark_data_smithi039_468595_object113:head# 2024-09-16T17:23:45.533 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2f98dea5:::benchmark_data_smithi039_468595_object283:head# 2024-09-16T17:23:45.533 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:30cb30ca:::benchmark_data_smithi039_468595_object20:head# 2024-09-16T17:23:45.534 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3197d42b:::benchmark_data_smithi039_468595_object185:head# 2024-09-16T17:23:45.534 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:31d63c7c:::benchmark_data_smithi039_468595_object208:head# 2024-09-16T17:23:45.535 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:31e9bbe0:::benchmark_data_smithi039_468595_object281:head# 2024-09-16T17:23:45.535 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:324e99cb:::benchmark_data_smithi039_468595_object251:head# 2024-09-16T17:23:45.536 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:340c903d:::benchmark_data_smithi039_468595_object213:head# 2024-09-16T17:23:45.536 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3477a114:::benchmark_data_smithi039_468595_object152:head# 2024-09-16T17:23:45.537 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:34dfc13e:::benchmark_data_smithi039_468595_object127:head# 2024-09-16T17:23:45.538 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:350ed3f6:::benchmark_data_smithi039_468595_object129:head# 2024-09-16T17:23:45.538 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3557128d:::benchmark_data_smithi039_468595_object248:head# 2024-09-16T17:23:45.538 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:372fe871:::benchmark_data_smithi039_468595_object270:head# 2024-09-16T17:23:45.539 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:37c7ec88:::benchmark_data_smithi039_468595_object201:head# 2024-09-16T17:23:45.540 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:38235478:::benchmark_data_smithi039_468595_object336:head# 2024-09-16T17:23:45.540 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:383bed4f:::benchmark_data_smithi039_468595_object293:head# 2024-09-16T17:23:45.540 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:38612daa:::benchmark_data_smithi039_468595_object90:head# 2024-09-16T17:23:45.541 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:392adff0:::benchmark_data_smithi039_468595_object95:head# 2024-09-16T17:23:45.541 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:39b8f791:::benchmark_data_smithi039_468595_object10:head# 2024-09-16T17:23:45.542 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:39c1b4a6:::benchmark_data_smithi039_468595_object21:head# 2024-09-16T17:23:45.542 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3a51e483:::benchmark_data_smithi039_468595_object250:head# 2024-09-16T17:23:45.543 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3a79852c:::benchmark_data_smithi039_468595_object230:head# 2024-09-16T17:23:45.543 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3ad5342b:::benchmark_data_smithi039_468595_object306:head# 2024-09-16T17:23:45.544 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3c00db33:::benchmark_data_smithi039_468595_object14:head# 2024-09-16T17:23:45.544 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3c0fb379:::benchmark_data_smithi039_468595_object271:head# 2024-09-16T17:23:45.545 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3c1b2ac1:::benchmark_data_smithi039_468595_object128:head# 2024-09-16T17:23:45.545 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3d12ed9d:::benchmark_data_smithi039_468595_object148:head# 2024-09-16T17:23:45.546 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3d61f832:::benchmark_data_smithi039_468595_object5:head# 2024-09-16T17:23:45.546 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3e7738a2:::benchmark_data_smithi039_468595_object63:head# 2024-09-16T17:23:45.547 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3ea7fca0:::benchmark_data_smithi039_468595_object334:head# 2024-09-16T17:23:45.547 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:40cb8af8:::benchmark_data_smithi039_468595_object279:head# 2024-09-16T17:23:45.547 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:427d927a:::benchmark_data_smithi039_468595_object59:head# 2024-09-16T17:23:45.548 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:43948a23:::benchmark_data_smithi039_468595_object72:head# 2024-09-16T17:23:45.548 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:44a75f96:::benchmark_data_smithi039_468595_object67:head# 2024-09-16T17:23:45.549 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:44b712dd:::benchmark_data_smithi039_468595_object37:head# 2024-09-16T17:23:45.549 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:455fc4b4:::benchmark_data_smithi039_468595_object326:head# 2024-09-16T17:23:45.549 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:457ff41f:::benchmark_data_smithi039_468595_object308:head# 2024-09-16T17:23:45.550 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:45b1ec39:::benchmark_data_smithi039_468595_object290:head# 2024-09-16T17:23:45.550 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:45d2fc81:::benchmark_data_smithi039_468595_object142:head# 2024-09-16T17:23:45.551 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4609546b:::benchmark_data_smithi039_468595_object79:head# 2024-09-16T17:23:45.551 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4720d590:::benchmark_data_smithi039_468595_object107:head# 2024-09-16T17:23:45.552 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:496e90ea:::benchmark_data_smithi039_468595_object110:head# 2024-09-16T17:23:45.552 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4a17d722:::benchmark_data_smithi039_468595_object202:head# 2024-09-16T17:23:45.552 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4a1b0efc:::benchmark_data_smithi039_468595_object32:head# 2024-09-16T17:23:45.553 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4a4bfa19:::benchmark_data_smithi039_468595_object22:head# 2024-09-16T17:23:45.553 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4b08ba44:::benchmark_data_smithi039_468595_object26:head# 2024-09-16T17:23:45.554 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4badec40:::benchmark_data_smithi039_468595_object85:head# 2024-09-16T17:23:45.554 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4bb4463a:::benchmark_data_smithi039_468595_object225:head# 2024-09-16T17:23:45.555 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4c6af759:::benchmark_data_smithi039_468595_object82:head# 2024-09-16T17:23:45.555 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4cec54db:::benchmark_data_smithi039_468595_object134:head# 2024-09-16T17:23:45.555 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4d5c4ae5:::benchmark_data_smithi039_468595_object253:head# 2024-09-16T17:23:45.556 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4d5f10b6:::benchmark_data_smithi039_468595_object328:head# 2024-09-16T17:23:45.556 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4d96ad59:::benchmark_data_smithi039_468595_object171:head# 2024-09-16T17:23:45.557 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4f1bbd9e:::benchmark_data_smithi039_468595_object68:head# 2024-09-16T17:23:45.557 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4ffc3aa2:::benchmark_data_smithi039_468595_object105:head# 2024-09-16T17:23:45.558 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:50903a7c:::benchmark_data_smithi039_468595_object233:head# 2024-09-16T17:23:45.558 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:50a0457c:::benchmark_data_smithi039_468595_object209:head# 2024-09-16T17:23:45.559 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:52207ac9:::benchmark_data_smithi039_468595_object124:head# 2024-09-16T17:23:45.560 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:52337313:::benchmark_data_smithi039_468595_object217:head# 2024-09-16T17:23:45.560 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:53e09036:::benchmark_data_smithi039_468595_object144:head# 2024-09-16T17:23:45.561 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5463a2e4:::benchmark_data_smithi039_468595_object307:head# 2024-09-16T17:23:45.562 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:54ba4047:::benchmark_data_smithi039_468595_object238:head# 2024-09-16T17:23:45.562 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:54e01bd5:::benchmark_data_smithi039_468595_object69:head# 2024-09-16T17:23:45.562 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:564adc8d:::benchmark_data_smithi039_468595_object197:head# 2024-09-16T17:23:45.563 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5687d21e:::benchmark_data_smithi039_468595_object91:head# 2024-09-16T17:23:45.563 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5707ca11:::benchmark_data_smithi039_468595_object11:head# 2024-09-16T17:23:45.564 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:57a103a8:::benchmark_data_smithi039_468595_object99:head# 2024-09-16T17:23:45.564 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:57e99a09:::benchmark_data_smithi039_468595_object199:head# 2024-09-16T17:23:45.565 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:585a6e4b:::benchmark_data_smithi039_468595_object104:head# 2024-09-16T17:23:45.565 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:59592b00:::benchmark_data_smithi039_468595_object168:head# 2024-09-16T17:23:45.566 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:598e986c:::benchmark_data_smithi039_468595_object237:head# 2024-09-16T17:23:45.566 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:59a63be9:::benchmark_data_smithi039_468595_object294:head# 2024-09-16T17:23:45.567 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5a6259af:::benchmark_data_smithi039_468595_object147:head# 2024-09-16T17:23:45.567 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5ab586e6:::benchmark_data_smithi039_468595_object28:head# 2024-09-16T17:23:45.568 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5bf7b5cb:::benchmark_data_smithi039_468595_object207:head# 2024-09-16T17:23:45.568 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5c5994f6:::benchmark_data_smithi039_468595_object157:head# 2024-09-16T17:23:45.569 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5cd70c81:::benchmark_data_smithi039_468595_object162:head# 2024-09-16T17:23:45.569 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5d78aa96:::benchmark_data_smithi039_468595_object302:head# 2024-09-16T17:23:45.570 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5d7f70d7:::benchmark_data_smithi039_468595_object60:head# 2024-09-16T17:23:45.570 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5df833aa:::benchmark_data_smithi039_468595_object284:head# 2024-09-16T17:23:45.571 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5ea7bc08:::benchmark_data_smithi039_468595_object92:head# 2024-09-16T17:23:45.571 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5ee3e58e:::benchmark_data_smithi039_468595_object223:head# 2024-09-16T17:23:45.572 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:62a8fa88:::benchmark_data_smithi039_468595_object56:head# 2024-09-16T17:23:45.572 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:62ee6725:::benchmark_data_smithi039_468595_object44:head# 2024-09-16T17:23:45.573 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:64ab38c2:::benchmark_data_smithi039_468595_object89:head# 2024-09-16T17:23:45.573 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6649a7f0:::benchmark_data_smithi039_468595_object240:head# 2024-09-16T17:23:45.574 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:66740cbf:::benchmark_data_smithi039_468595_object215:head# 2024-09-16T17:23:45.574 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6d6ec100:::benchmark_data_smithi039_468595_object324:head# 2024-09-16T17:23:45.575 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6d7e0208:::benchmark_data_smithi039_468595_object143:head# 2024-09-16T17:23:45.575 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6dc530d1:::benchmark_data_smithi039_468595_object274:head# 2024-09-16T17:23:45.576 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6e432b0c:::benchmark_data_smithi039_468595_object333:head# 2024-09-16T17:23:45.576 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6ea23d44:::benchmark_data_smithi039_468595_object259:head# 2024-09-16T17:23:45.577 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6edc8698:::benchmark_data_smithi039_468595_object299:head# 2024-09-16T17:23:45.577 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6ef8850a:::benchmark_data_smithi039_468595_object112:head# 2024-09-16T17:23:45.577 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6fcac29e:::benchmark_data_smithi039_468595_object46:head# 2024-09-16T17:23:45.578 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:703dce8c:::benchmark_data_smithi039_468595_object303:head# 2024-09-16T17:23:45.578 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:714274ce:::benchmark_data_smithi039_468595_object98:head# 2024-09-16T17:23:45.578 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:721fa733:::benchmark_data_smithi039_468595_object332:head# 2024-09-16T17:23:45.579 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:722f4cb7:::benchmark_data_smithi039_468595_object219:head# 2024-09-16T17:23:45.579 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:72ea8d9f:::benchmark_data_smithi039_468595_object87:head# 2024-09-16T17:23:45.580 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:73f7fc61:::benchmark_data_smithi039_468595_object4:head# 2024-09-16T17:23:45.580 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:73fcfe0e:::benchmark_data_smithi039_468595_object205:head# 2024-09-16T17:23:45.581 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:754c286d:::benchmark_data_smithi039_468595_object41:head# 2024-09-16T17:23:45.581 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:768427e5:::benchmark_data_smithi039_468595_object260:head# 2024-09-16T17:23:45.582 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7685c881:::benchmark_data_smithi039_468595_object123:head# 2024-09-16T17:23:45.582 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:77e63c42:::benchmark_data_smithi039_468595_object231:head# 2024-09-16T17:23:45.583 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:790cd8e2:::benchmark_data_smithi039_468595_object19:head# 2024-09-16T17:23:45.583 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7baf820d:::benchmark_data_smithi039_468595_object159:head# 2024-09-16T17:23:45.584 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7cafc3a6:::benchmark_data_smithi039_468595_object38:head# 2024-09-16T17:23:45.584 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7f14319c:::benchmark_data_smithi039_468595_object245:head# 2024-09-16T17:23:45.585 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:80918c14:::benchmark_data_smithi039_468595_object16:head# 2024-09-16T17:23:45.586 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:80d9f722:::benchmark_data_smithi039_468595_object106:head# 2024-09-16T17:23:45.586 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:80fce290:::benchmark_data_smithi039_468595_object184:head# 2024-09-16T17:23:45.587 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:81f53868:::benchmark_data_smithi039_468595_object298:head# 2024-09-16T17:23:45.587 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:844f9018:::benchmark_data_smithi039_468595_object262:head# 2024-09-16T17:23:45.587 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:84bb7162:::benchmark_data_smithi039_468595_object311:head# 2024-09-16T17:23:45.588 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:850b1a92:::benchmark_data_smithi039_468595_object138:head# 2024-09-16T17:23:45.588 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8569d272:::benchmark_data_smithi039_468595_object241:head# 2024-09-16T17:23:45.589 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:863caf4a:::benchmark_data_smithi039_468595_object330:head# 2024-09-16T17:23:45.590 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:869f7a6a:::benchmark_data_smithi039_468595_object265:head# 2024-09-16T17:23:45.590 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8701fee5:::benchmark_data_smithi039_468595_object135:head# 2024-09-16T17:23:45.590 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:880a2a14:::benchmark_data_smithi039_468595_object52:head# 2024-09-16T17:23:45.591 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:887b6945:::benchmark_data_smithi039_468595_object141:head# 2024-09-16T17:23:45.591 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:88fcacfa:::benchmark_data_smithi039_468595_object25:head# 2024-09-16T17:23:45.592 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:891807e1:::benchmark_data_smithi039_468595_object114:head# 2024-09-16T17:23:45.592 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:89d1091d:::benchmark_data_smithi039_468595_object0:head# 2024-09-16T17:23:45.593 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8b43ece7:::benchmark_data_smithi039_468595_object81:head# 2024-09-16T17:23:45.593 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8dace154:::benchmark_data_smithi039_468595_object173:head# 2024-09-16T17:23:45.594 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8de5b1c0:::benchmark_data_smithi039_468595_object163:head# 2024-09-16T17:23:45.594 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8e6fcc3f:::benchmark_data_smithi039_468595_object29:head# 2024-09-16T17:23:45.595 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8f0ce81c:::benchmark_data_smithi039_468595_object94:head# 2024-09-16T17:23:45.595 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8fcb7113:::benchmark_data_smithi039_468595_object266:head# 2024-09-16T17:23:45.595 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9045a82d:::benchmark_data_smithi039_468595_object43:head# 2024-09-16T17:23:45.596 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:91f2e0f9:::benchmark_data_smithi039_468595_object64:head# 2024-09-16T17:23:45.597 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9237253d:::benchmark_data_smithi039_468595_object329:head# 2024-09-16T17:23:45.597 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:926c2408:::benchmark_data_smithi039_468595_object301:head# 2024-09-16T17:23:45.597 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92897a94:::benchmark_data_smithi039_468595_object156:head# 2024-09-16T17:23:45.598 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92c8bc05:::benchmark_data_smithi039_468595_object97:head# 2024-09-16T17:23:45.598 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:93531de3:::benchmark_data_smithi039_468595_object78:head# 2024-09-16T17:23:45.598 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:97a77275:::benchmark_data_smithi039_468595_object242:head# 2024-09-16T17:23:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:97b3ce3c:::benchmark_data_smithi039_468595_object305:head# 2024-09-16T17:23:45.599 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:981dc61a:::benchmark_data_smithi039_468595_object195:head# 2024-09-16T17:23:45.600 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:98aa50c6:::benchmark_data_smithi039_468595_object276:head# 2024-09-16T17:23:45.601 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:99e59096:::benchmark_data_smithi039_468595_object161:head# 2024-09-16T17:23:45.601 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:99ed60a7:::benchmark_data_smithi039_468595_object17:head# 2024-09-16T17:23:45.602 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9a14b9a4:::benchmark_data_smithi039_468595_object50:head# 2024-09-16T17:23:45.602 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9b038352:::benchmark_data_smithi039_468595_object224:head# 2024-09-16T17:23:45.602 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9b1f346f:::benchmark_data_smithi039_468595_object30:head# 2024-09-16T17:23:45.603 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9b63c8c4:::benchmark_data_smithi039_468595_object318:head# 2024-09-16T17:23:45.603 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9bc8111e:::benchmark_data_smithi039_468595_object212:head# 2024-09-16T17:23:45.604 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9c275498:::benchmark_data_smithi039_468595_object297:head# 2024-09-16T17:23:45.604 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9c2f4e7a:::benchmark_data_smithi039_468595_object58:head# 2024-09-16T17:23:45.605 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9cfeb30a:::benchmark_data_smithi039_468595_object316:head# 2024-09-16T17:23:45.605 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9e572ace:::benchmark_data_smithi039_468595_object125:head# 2024-09-16T17:23:45.606 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9fe9f39f:::benchmark_data_smithi039_468595_object335:head# 2024-09-16T17:23:45.606 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a1aa6f8e:::benchmark_data_smithi039_468595_object291:head# 2024-09-16T17:23:45.606 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a1fbc247:::benchmark_data_smithi039_468595_object186:head# 2024-09-16T17:23:45.607 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a217378b:::benchmark_data_smithi039_468595_object228:head# 2024-09-16T17:23:45.607 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a25c34de:::benchmark_data_smithi039_468595_object182:head# 2024-09-16T17:23:45.608 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a376bc42:::benchmark_data_smithi039_468595_object309:head# 2024-09-16T17:23:45.609 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a4c17a67:::benchmark_data_smithi039_468595_object187:head# 2024-09-16T17:23:45.609 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a7a04742:::benchmark_data_smithi039_468595_object317:head# 2024-09-16T17:23:45.609 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a8eb188c:::benchmark_data_smithi039_468595_object153:head# 2024-09-16T17:23:45.610 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a936fd24:::benchmark_data_smithi039_468595_object76:head# 2024-09-16T17:23:45.610 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ac3170c8:::benchmark_data_smithi039_468595_object146:head# 2024-09-16T17:23:45.610 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ace79a8c:::benchmark_data_smithi039_468595_object48:head# 2024-09-16T17:23:45.611 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ad369937:::benchmark_data_smithi039_468595_object188:head# 2024-09-16T17:23:45.612 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:aeca547f:::benchmark_data_smithi039_468595_object200:head# 2024-09-16T17:23:45.612 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:aed81ba6:::benchmark_data_smithi039_468595_object15:head# 2024-09-16T17:23:45.612 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b0201ed8:::benchmark_data_smithi039_468595_object211:head# 2024-09-16T17:23:45.613 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b0f1c022:::benchmark_data_smithi039_468595_object252:head# 2024-09-16T17:23:45.613 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b1e5f2ad:::benchmark_data_smithi039_468595_object18:head# 2024-09-16T17:23:45.614 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b2d0bf87:::benchmark_data_smithi039_468595_object120:head# 2024-09-16T17:23:45.614 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b344ddc1:::benchmark_data_smithi039_468595_object36:head# 2024-09-16T17:23:45.615 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b357dd0c:::benchmark_data_smithi039_468595_object39:head# 2024-09-16T17:23:45.615 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b46a710e:::benchmark_data_smithi039_468595_object164:head# 2024-09-16T17:23:45.616 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b57c02fe:::benchmark_data_smithi039_468595_object285:head# 2024-09-16T17:23:45.616 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b5878bb3:::benchmark_data_smithi039_468595_object254:head# 2024-09-16T17:23:45.617 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b601fe0a:::benchmark_data_smithi039_468595_object57:head# 2024-09-16T17:23:45.617 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b67d6553:::benchmark_data_smithi039_468595_object100:head# 2024-09-16T17:23:45.618 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b722fd33:::benchmark_data_smithi039_468595_object282:head# 2024-09-16T17:23:45.618 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b7a78853:::benchmark_data_smithi039_468595_object66:head# 2024-09-16T17:23:45.618 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b8163b8e:::benchmark_data_smithi039_468595_object176:head# 2024-09-16T17:23:45.619 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-09-16T17:23:45.619 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b885a7d2:::benchmark_data_smithi039_468595_object166:head# 2024-09-16T17:23:45.620 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b8d3729a:::benchmark_data_smithi039_468595_object268:head# 2024-09-16T17:23:45.621 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bbafc8a3:::benchmark_data_smithi039_468595_object111:head# 2024-09-16T17:23:45.621 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bbd3b87a:::benchmark_data_smithi039_468595_object145:head# 2024-09-16T17:23:45.622 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bc22f79f:::benchmark_data_smithi039_468595_object189:head# 2024-09-16T17:23:45.622 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bd7ee7ea:::benchmark_data_smithi039_468595_object235:head# 2024-09-16T17:23:45.623 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bd9dabc0:::benchmark_data_smithi039_468595_object220:head# 2024-09-16T17:23:45.623 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bdd0883c:::benchmark_data_smithi039_468595_object3:head# 2024-09-16T17:23:45.624 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:beb7571d:::benchmark_data_smithi039_468595_object49:head# 2024-09-16T17:23:45.624 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c0a2001b:::benchmark_data_smithi039_468595_object244:head# 2024-09-16T17:23:45.625 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c0e305af:::benchmark_data_smithi039_468595_object288:head# 2024-09-16T17:23:45.625 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c1a6dd82:::benchmark_data_smithi039_468595_object132:head# 2024-09-16T17:23:45.625 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c3460b41:::benchmark_data_smithi039_468595_object175:head# 2024-09-16T17:23:45.626 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c3879720:::benchmark_data_smithi039_468595_object47:head# 2024-09-16T17:23:45.626 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c57a1706:::benchmark_data_smithi039_468595_object179:head# 2024-09-16T17:23:45.627 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c58f9a8e:::benchmark_data_smithi039_468595_object257:head# 2024-09-16T17:23:45.627 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c8eedfc1:::benchmark_data_smithi039_468595_object174:head# 2024-09-16T17:23:45.628 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c99442a0:::benchmark_data_smithi039_468595_object321:head# 2024-09-16T17:23:45.628 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ca33a770:::benchmark_data_smithi039_468595_object210:head# 2024-09-16T17:23:45.629 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cb34b9ee:::benchmark_data_smithi039_468595_object55:head# 2024-09-16T17:23:45.629 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cb7004f2:::benchmark_data_smithi039_468595_object167:head# 2024-09-16T17:23:45.630 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ce600566:::benchmark_data_smithi039_468595_object126:head# 2024-09-16T17:23:45.630 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cf33d8ae:::benchmark_data_smithi039_468595_object62:head# 2024-09-16T17:23:45.631 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cf3bb6d3:::benchmark_data_smithi039_468595_object118:head# 2024-09-16T17:23:45.631 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cfbe5e78:::benchmark_data_smithi039_468595_object103:head# 2024-09-16T17:23:45.632 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d07a3480:::benchmark_data_smithi039_468595_object165:head# 2024-09-16T17:23:45.632 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d0bfbe29:::benchmark_data_smithi039_468595_object12:head# 2024-09-16T17:23:45.633 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d0f9d51d:::benchmark_data_smithi039_468595_object194:head# 2024-09-16T17:23:45.633 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d289528f:::benchmark_data_smithi039_468595_object264:head# 2024-09-16T17:23:45.634 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d30255de:::benchmark_data_smithi039_468595_object226:head# 2024-09-16T17:23:45.634 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d3184e15:::benchmark_data_smithi039_468595_object73:head# 2024-09-16T17:23:45.634 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d351bdaf:::benchmark_data_smithi039_468595_object249:head# 2024-09-16T17:23:45.635 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d47ba051:::benchmark_data_smithi039_468595_object121:head# 2024-09-16T17:23:45.636 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d4936f9d:::benchmark_data_smithi039_468595_object115:head# 2024-09-16T17:23:45.636 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d4bd7d38:::benchmark_data_smithi039_468595_object278:head# 2024-09-16T17:23:45.637 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d4be707a:::benchmark_data_smithi039_468595_object273:head# 2024-09-16T17:23:45.637 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d52ca634:::benchmark_data_smithi039_468595_object151:head# 2024-09-16T17:23:45.638 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d7dfb1a9:::benchmark_data_smithi039_468595_object150:head# 2024-09-16T17:23:45.638 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d7eb6c60:::benchmark_data_smithi039_468595_object88:head# 2024-09-16T17:23:45.639 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d8a669c3:::benchmark_data_smithi039_468595_object191:head# 2024-09-16T17:23:45.639 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dbcf17ab:::benchmark_data_smithi039_468595_object322:head# 2024-09-16T17:23:45.640 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dbf09c2f:::benchmark_data_smithi039_468595_object221:head# 2024-09-16T17:23:45.640 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dc452c3b:::benchmark_data_smithi039_468595_object13:head# 2024-09-16T17:23:45.641 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dc809645:::benchmark_data_smithi039_468595_object27:head# 2024-09-16T17:23:45.641 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dcf39b6c:::benchmark_data_smithi039_468595_object310:head# 2024-09-16T17:23:45.641 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ddb3c124:::benchmark_data_smithi039_468595_object154:head# 2024-09-16T17:23:45.642 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:de15f6ae:::benchmark_data_smithi039_468595_object327:head# 2024-09-16T17:23:45.642 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:de2003b5:::benchmark_data_smithi039_468595_object71:head# 2024-09-16T17:23:45.643 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:df17aa0c:::benchmark_data_smithi039_468595_object53:head# 2024-09-16T17:23:45.643 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:df436e46:::benchmark_data_smithi039_468595_object277:head# 2024-09-16T17:23:45.644 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e0539c89:::benchmark_data_smithi039_468595_object246:head# 2024-09-16T17:23:45.644 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e09fbc15:::benchmark_data_smithi039_468595_object181:head# 2024-09-16T17:23:45.645 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e1f04201:::benchmark_data_smithi039_468595_object80:head# 2024-09-16T17:23:45.645 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e3ab9a7b:::benchmark_data_smithi039_468595_object315:head# 2024-09-16T17:23:45.646 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e3d55e4c:::benchmark_data_smithi039_468595_object255:head# 2024-09-16T17:23:45.647 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e3ee6c37:::benchmark_data_smithi039_468595_object287:head# 2024-09-16T17:23:45.647 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e4fd5aea:::benchmark_data_smithi039_468595_object331:head# 2024-09-16T17:23:45.647 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e595dc31:::benchmark_data_smithi039_468595_object193:head# 2024-09-16T17:23:45.648 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e6d657ff:::benchmark_data_smithi039_468595_object313:head# 2024-09-16T17:23:45.648 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e7288009:::benchmark_data_smithi039_468595_object160:head# 2024-09-16T17:23:45.649 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e781cf57:::benchmark_data_smithi039_468595_object272:head# 2024-09-16T17:23:45.649 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e7a5c1a0:::benchmark_data_smithi039_468595_object137:head# 2024-09-16T17:23:45.650 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e7ae1cf0:::benchmark_data_smithi039_468595_object203:head# 2024-09-16T17:23:45.650 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e8c79fcc:::benchmark_data_smithi039_468595_object65:head# 2024-09-16T17:23:45.651 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e9c1a181:::benchmark_data_smithi039_468595_object40:head# 2024-09-16T17:23:45.651 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ea98c336:::benchmark_data_smithi039_468595_object77:head# 2024-09-16T17:23:45.652 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:eb115269:::benchmark_data_smithi039_468595_object319:head# 2024-09-16T17:23:45.652 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:eb1e19f4:::benchmark_data_smithi039_468595_object177:head# 2024-09-16T17:23:45.652 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:eb9c8a80:::benchmark_data_smithi039_468595_object300:head# 2024-09-16T17:23:45.656 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ebbadbe5:::benchmark_data_smithi039_468595_object158:head# 2024-09-16T17:23:45.657 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ec76a88e:::benchmark_data_smithi039_468595_object23:head# 2024-09-16T17:23:45.657 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:eca7d2be:::benchmark_data_smithi039_468595_object169:head# 2024-09-16T17:23:45.657 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ed0424ec:::benchmark_data_smithi039_468595_object289:head# 2024-09-16T17:23:45.657 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:edfb9ce3:::benchmark_data_smithi039_468595_object216:head# 2024-09-16T17:23:45.658 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ee55e927:::benchmark_data_smithi039_468595_object155:head# 2024-09-16T17:23:45.659 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ef41dc71:::benchmark_data_smithi039_468595_object139:head# 2024-09-16T17:23:45.659 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ef47a94f:::benchmark_data_smithi039_468595_object130:head# 2024-09-16T17:23:45.659 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:efa9c4c9:::benchmark_data_smithi039_468595_object206:head# 2024-09-16T17:23:45.659 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f080017e:::benchmark_data_smithi039_468595_object86:head# 2024-09-16T17:23:45.660 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f13659e9:::benchmark_data_smithi039_468595_object304:head# 2024-09-16T17:23:45.660 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f3269939:::benchmark_data_smithi039_468595_object83:head# 2024-09-16T17:23:45.660 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f4549a74:::benchmark_data_smithi039_468595_object314:head# 2024-09-16T17:23:45.660 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f582b90f:::benchmark_data_smithi039_468595_object198:head# 2024-09-16T17:23:45.660 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f62d4d3c:::benchmark_data_smithi039_468595_object227:head# 2024-09-16T17:23:45.661 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f750af9a:::benchmark_data_smithi039_468595_object35:head# 2024-09-16T17:23:45.661 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f7fa6095:::benchmark_data_smithi039_468595_object117:head# 2024-09-16T17:23:45.661 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f801d8a3:::benchmark_data_smithi039_468595_object267:head# 2024-09-16T17:23:45.661 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f8ed550b:::benchmark_data_smithi039_468595_object149:head# 2024-09-16T17:23:45.662 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f910a5fe:::benchmark_data_smithi039_468595_object192:head# 2024-09-16T17:23:45.662 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f910f5db:::benchmark_data_smithi039_468595_object84:head# 2024-09-16T17:23:45.662 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f94d74ec:::benchmark_data_smithi039_468595_object31:head# 2024-09-16T17:23:45.663 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f9debaaf:::benchmark_data_smithi039_468595_object320:head# 2024-09-16T17:23:45.663 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fa5fc20d:::benchmark_data_smithi039_468595_object275:head# 2024-09-16T17:23:45.664 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fb817c6e:::benchmark_data_smithi039_468595_object119:head# 2024-09-16T17:23:45.664 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fbc18c31:::benchmark_data_smithi039_468595_object8:head# 2024-09-16T17:23:45.664 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fc0d8571:::benchmark_data_smithi039_468595_object93:head# 2024-09-16T17:23:45.665 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fc0f46c2:::benchmark_data_smithi039_468595_object190:head# 2024-09-16T17:23:45.665 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fc0f9e7a:::benchmark_data_smithi039_468595_object312:head# 2024-09-16T17:23:45.666 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fe77dedf:::benchmark_data_smithi039_468595_object296:head# 2024-09-16T17:23:45.666 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:feca5f29:::benchmark_data_smithi039_468595_object33:head# 2024-09-16T17:23:45.667 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fef32071:::benchmark_data_smithi039_468595_object42:head# 2024-09-16T17:23:45.667 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ff290914:::benchmark_data_smithi039_468595_object133:head# 2024-09-16T17:23:45.667 INFO:tasks.workunit.client.0.smithi039.stderr:Export successful 2024-09-16T17:23:45.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:175: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T17:23:45.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-16T17:23:45.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T17:23:45.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:23:45.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:23:45.957 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:23:45.957 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:23:45.957 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:23:45.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:23:45.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:23:45.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:23:45.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-16T17:23:45.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-16T17:23:45.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:23:45.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:23:45.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:23:45.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:23:45.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T17:23:45.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T17:23:45.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-16T17:23:45.959 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:23:45.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T17:23:45.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:23:45.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-16T17:23:45.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T17:23:45.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T17:23:45.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T17:23:45.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T17:23:45.997 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:23:45.995+0000 7fa0e347e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:23:46.000 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:23:45.999+0000 7fa0e347e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:23:46.002 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:23:45.999+0000 7fa0e347e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:23:46.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T17:23:46.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:23:46.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:23:46.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:23:46.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:23:46.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:46.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:23:46.407 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:23:46.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:46.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:46.548 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:23:46.547+0000 7fa0e347e7c0 -1 Falling back to public interface 2024-09-16T17:23:46.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:47.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:47.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:47.835 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:23:47.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:23:47.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:47.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:48.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:23:49.096 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:23:49.095+0000 7fa0e347e7c0 -1 osd.0 9 log_to_monitors true 2024-09-16T17:23:49.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:23:49.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:23:49.311 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:23:49.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:23:49.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:23:49.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:23:49.776 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 12 up_thru 7 down_at 10 last_clean_interval [5,9) [v2:127.0.0.1:6802/1726351071,v1:127.0.0.1:6803/1726351071] [v2:127.0.0.1:6804/1726351071,v1:127.0.0.1:6805/1726351071] exists,up c1945d7e-0f4d-49d8-91bb-17a1555e217a 2024-09-16T17:23:49.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:23:49.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:23:49.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:23:49.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:177: TEST_import_after_split: ceph osd pool set foo pg_num 2 2024-09-16T17:23:50.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:178: TEST_import_after_split: sleep 5 2024-09-16T17:23:55.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: ceph daemon osd.0 perf dump 2024-09-16T17:23:55.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: jq .osd.numpg 2024-09-16T17:23:55.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: grep 1 2024-09-16T17:23:55.909 INFO:tasks.workunit.client.0.smithi039.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-16T17:23:55.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:180: TEST_import_after_split: wait_for_clean 2024-09-16T17:23:55.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:23:55.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:23:55.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:23:55.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:23:55.912 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:23:55.912 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:23:55.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:23:55.912 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:23:55.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:23:56.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:23:56.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:23:56.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:23:56.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:23:56.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:23:56.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:23:56.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T17:23:56.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:23:56.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:23:56.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:23:56.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607556 2024-09-16T17:23:56.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607556 2024-09-16T17:23:56.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-51539607556' 2024-09-16T17:23:56.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:23:56.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-51539607556 2024-09-16T17:23:56.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:23:56.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:23:56.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:23:56.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-51539607556 2024-09-16T17:23:56.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607556 2024-09-16T17:23:56.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 51539607556' 2024-09-16T17:23:56.795 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 51539607556 2024-09-16T17:23:56.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:23:57.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607554 -lt 51539607556 2024-09-16T17:23:57.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:23:58.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:23:58.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:23:58.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607556 -lt 51539607556 2024-09-16T17:23:58.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:23:58.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:23:58.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:23:59.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-16T17:23:59.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:23:59.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:23:59.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:23:59.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:23:59.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:23:59.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:23:59.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:23:59.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T17:23:59.748 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:23:59.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:23:59.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:24:00.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-16T17:24:00.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:24:00.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:24:00.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:182: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-09-16T17:24:00.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:24:00.351 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:24:00.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:24:00.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:24:00.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:24:00.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:24:00.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:184: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-09-16T17:24:01.836 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-16T17:24:01.836 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-16T17:24:01.836 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-16T17:24:02.128 INFO:tasks.workunit.client.0.smithi039.stdout:Remove successful 2024-09-16T17:24:02.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:187: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-16T17:24:03.620 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-09-16T17:24:03.620 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.0 2024-09-16T17:24:03.621 INFO:tasks.workunit.client.0.smithi039.stdout:pool 1 pg_num 1 -> 2 2024-09-16T17:24:03.621 INFO:tasks.workunit.client.0.smithi039.stderr: children are 1.1 2024-09-16T17:24:03.621 INFO:tasks.workunit.client.0.smithi039.stderr:pgid 1.0 splits to 1.1 and 1.1 exists 2024-09-16T17:24:03.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:189: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-09-16T17:24:04.619 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-16T17:24:04.619 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-16T17:24:04.619 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-09-16T17:24:04.901 INFO:tasks.workunit.client.0.smithi039.stdout:Remove successful 2024-09-16T17:24:05.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:191: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-16T17:24:06.354 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-09-16T17:24:06.354 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.0 2024-09-16T17:24:06.355 INFO:tasks.workunit.client.0.smithi039.stdout:pool 1 pg_num 1 -> 2 2024-09-16T17:24:06.355 INFO:tasks.workunit.client.0.smithi039.stderr: children are 1.1 2024-09-16T17:24:06.355 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:000564e5:::benchmark_data_smithi039_468595_object239:head# 2024-09-16T17:24:06.355 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.504 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:006f988d:::benchmark_data_smithi039_468595_object234:head# 2024-09-16T17:24:06.504 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.537 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:00e12f2f:::benchmark_data_smithi039_468595_object131:head# 2024-09-16T17:24:06.537 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.571 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:028910e9:::benchmark_data_smithi039_468595_object24:head# 2024-09-16T17:24:06.571 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.604 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:056c3ba9:::benchmark_data_smithi039_468595_object172:head# 2024-09-16T17:24:06.604 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.638 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:05c59347:::benchmark_data_smithi039_468595_object2:head# 2024-09-16T17:24:06.638 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.671 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:06eba0d7:::benchmark_data_smithi039_468595_object45:head# 2024-09-16T17:24:06.671 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.730 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0751acb2:::benchmark_data_smithi039_468595_object183:head# 2024-09-16T17:24:06.730 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.772 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:07c393fb:::benchmark_data_smithi039_468595_object136:head# 2024-09-16T17:24:06.772 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.805 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0a3ad74d:::benchmark_data_smithi039_468595_object232:head# 2024-09-16T17:24:06.805 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.839 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0a7cf200:::benchmark_data_smithi039_468595_object214:head# 2024-09-16T17:24:06.839 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.872 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0bd67d89:::benchmark_data_smithi039_468595_object247:head# 2024-09-16T17:24:06.872 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0c3f0015:::benchmark_data_smithi039_468595_object96:head# 2024-09-16T17:24:06.905 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0c4ca308:::benchmark_data_smithi039_468595_object122:head# 2024-09-16T17:24:06.939 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0c96026c:::benchmark_data_smithi039_468595_object102:head# 2024-09-16T17:24:06.972 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.006 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0d76d630:::benchmark_data_smithi039_468595_object34:head# 2024-09-16T17:24:07.006 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.039 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0e30d34f:::benchmark_data_smithi039_468595_object218:head# 2024-09-16T17:24:07.040 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.073 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:11a6ffc3:::benchmark_data_smithi039_468595_object70:head# 2024-09-16T17:24:07.073 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.106 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:13470305:::benchmark_data_smithi039_468595_object140:head# 2024-09-16T17:24:07.106 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.140 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:143bf9f0:::benchmark_data_smithi039_468595_object9:head# 2024-09-16T17:24:07.140 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.173 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:147e98f9:::benchmark_data_smithi039_468595_object51:head# 2024-09-16T17:24:07.173 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.207 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:14ee031d:::benchmark_data_smithi039_468595_object325:head# 2024-09-16T17:24:07.207 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.240 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1569fe73:::benchmark_data_smithi039_468595_object101:head# 2024-09-16T17:24:07.240 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.273 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:157740f9:::benchmark_data_smithi039_468595_object292:head# 2024-09-16T17:24:07.274 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.307 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:15c58c9e:::benchmark_data_smithi039_468595_object323:head# 2024-09-16T17:24:07.307 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.340 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:15cba213:::benchmark_data_smithi039_468595_object75:head# 2024-09-16T17:24:07.341 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.374 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1682914d:::benchmark_data_smithi039_468595_object1:head# 2024-09-16T17:24:07.374 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.407 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:16ca303c:::benchmark_data_smithi039_468595_object236:head# 2024-09-16T17:24:07.407 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.441 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:170afdae:::benchmark_data_smithi039_468595_object116:head# 2024-09-16T17:24:07.441 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.474 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:18f19464:::benchmark_data_smithi039_468595_object258:head# 2024-09-16T17:24:07.474 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.508 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1a549628:::benchmark_data_smithi039_468595_object54:head# 2024-09-16T17:24:07.508 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.550 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1b3855f6:::benchmark_data_smithi039_468595_object256:head# 2024-09-16T17:24:07.550 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.583 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1b555b63:::benchmark_data_smithi039_468595_object178:head# 2024-09-16T17:24:07.583 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.617 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1cbd37a8:::benchmark_data_smithi039_468595_object269:head# 2024-09-16T17:24:07.617 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.650 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1cc2b390:::benchmark_data_smithi039_468595_object295:head# 2024-09-16T17:24:07.650 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.684 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1dd3e2de:::benchmark_data_smithi039_468595_object108:head# 2024-09-16T17:24:07.684 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.717 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:20866bca:::benchmark_data_smithi039_468595_object6:head# 2024-09-16T17:24:07.717 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.751 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:20e66408:::benchmark_data_smithi039_468595_object74:head# 2024-09-16T17:24:07.751 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.784 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2223a6bd:::benchmark_data_smithi039_468595_object7:head# 2024-09-16T17:24:07.784 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.817 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:22b0abf3:::benchmark_data_smithi039_468595_object229:head# 2024-09-16T17:24:07.818 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.851 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:22d6afb2:::benchmark_data_smithi039_468595_object243:head# 2024-09-16T17:24:07.851 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.884 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2474db63:::benchmark_data_smithi039_468595_object61:head# 2024-09-16T17:24:07.884 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.918 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:267ac8e8:::benchmark_data_smithi039_468595_object280:head# 2024-09-16T17:24:07.918 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.951 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2716b92a:::benchmark_data_smithi039_468595_object196:head# 2024-09-16T17:24:07.951 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:07.985 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:275da9db:::benchmark_data_smithi039_468595_object261:head# 2024-09-16T17:24:07.985 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.018 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2792d397:::benchmark_data_smithi039_468595_object286:head# 2024-09-16T17:24:08.018 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.052 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:28dc7dc3:::benchmark_data_smithi039_468595_object204:head# 2024-09-16T17:24:08.052 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.119 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2b30d01d:::benchmark_data_smithi039_468595_object170:head# 2024-09-16T17:24:08.119 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.169 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2b359d12:::benchmark_data_smithi039_468595_object109:head# 2024-09-16T17:24:08.169 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.202 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2bc2d150:::benchmark_data_smithi039_468595_object263:head# 2024-09-16T17:24:08.202 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.244 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2c298f0c:::benchmark_data_smithi039_468595_object222:head# 2024-09-16T17:24:08.244 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.294 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2dfe2ce0:::benchmark_data_smithi039_468595_object180:head# 2024-09-16T17:24:08.294 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.344 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2f3a15aa:::benchmark_data_smithi039_468595_object113:head# 2024-09-16T17:24:08.344 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.420 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2f98dea5:::benchmark_data_smithi039_468595_object283:head# 2024-09-16T17:24:08.420 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.487 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:30cb30ca:::benchmark_data_smithi039_468595_object20:head# 2024-09-16T17:24:08.487 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.537 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3197d42b:::benchmark_data_smithi039_468595_object185:head# 2024-09-16T17:24:08.537 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.579 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:31d63c7c:::benchmark_data_smithi039_468595_object208:head# 2024-09-16T17:24:08.579 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.621 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:31e9bbe0:::benchmark_data_smithi039_468595_object281:head# 2024-09-16T17:24:08.621 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.663 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:324e99cb:::benchmark_data_smithi039_468595_object251:head# 2024-09-16T17:24:08.663 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.705 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:340c903d:::benchmark_data_smithi039_468595_object213:head# 2024-09-16T17:24:08.705 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.738 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3477a114:::benchmark_data_smithi039_468595_object152:head# 2024-09-16T17:24:08.738 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.771 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:34dfc13e:::benchmark_data_smithi039_468595_object127:head# 2024-09-16T17:24:08.771 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.805 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:350ed3f6:::benchmark_data_smithi039_468595_object129:head# 2024-09-16T17:24:08.805 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.838 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3557128d:::benchmark_data_smithi039_468595_object248:head# 2024-09-16T17:24:08.838 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.872 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:372fe871:::benchmark_data_smithi039_468595_object270:head# 2024-09-16T17:24:08.872 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.914 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:37c7ec88:::benchmark_data_smithi039_468595_object201:head# 2024-09-16T17:24:08.914 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.947 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:38235478:::benchmark_data_smithi039_468595_object336:head# 2024-09-16T17:24:08.947 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:08.981 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:383bed4f:::benchmark_data_smithi039_468595_object293:head# 2024-09-16T17:24:08.981 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.014 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:38612daa:::benchmark_data_smithi039_468595_object90:head# 2024-09-16T17:24:09.014 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.047 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:392adff0:::benchmark_data_smithi039_468595_object95:head# 2024-09-16T17:24:09.047 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.081 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:39b8f791:::benchmark_data_smithi039_468595_object10:head# 2024-09-16T17:24:09.081 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.114 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:39c1b4a6:::benchmark_data_smithi039_468595_object21:head# 2024-09-16T17:24:09.115 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.148 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3a51e483:::benchmark_data_smithi039_468595_object250:head# 2024-09-16T17:24:09.148 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.181 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3a79852c:::benchmark_data_smithi039_468595_object230:head# 2024-09-16T17:24:09.181 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.215 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3ad5342b:::benchmark_data_smithi039_468595_object306:head# 2024-09-16T17:24:09.215 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.248 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3c00db33:::benchmark_data_smithi039_468595_object14:head# 2024-09-16T17:24:09.249 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.282 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3c0fb379:::benchmark_data_smithi039_468595_object271:head# 2024-09-16T17:24:09.282 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.315 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3c1b2ac1:::benchmark_data_smithi039_468595_object128:head# 2024-09-16T17:24:09.315 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.349 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3d12ed9d:::benchmark_data_smithi039_468595_object148:head# 2024-09-16T17:24:09.349 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.382 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3d61f832:::benchmark_data_smithi039_468595_object5:head# 2024-09-16T17:24:09.382 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.416 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3e7738a2:::benchmark_data_smithi039_468595_object63:head# 2024-09-16T17:24:09.416 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.449 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3ea7fca0:::benchmark_data_smithi039_468595_object334:head# 2024-09-16T17:24:09.449 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.483 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:40cb8af8:::benchmark_data_smithi039_468595_object279:head# 2024-09-16T17:24:09.483 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.516 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:427d927a:::benchmark_data_smithi039_468595_object59:head# 2024-09-16T17:24:09.516 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.549 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:43948a23:::benchmark_data_smithi039_468595_object72:head# 2024-09-16T17:24:09.550 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.591 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:44a75f96:::benchmark_data_smithi039_468595_object67:head# 2024-09-16T17:24:09.592 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.625 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:44b712dd:::benchmark_data_smithi039_468595_object37:head# 2024-09-16T17:24:09.625 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.658 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:455fc4b4:::benchmark_data_smithi039_468595_object326:head# 2024-09-16T17:24:09.659 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.692 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:457ff41f:::benchmark_data_smithi039_468595_object308:head# 2024-09-16T17:24:09.692 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.725 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:45b1ec39:::benchmark_data_smithi039_468595_object290:head# 2024-09-16T17:24:09.725 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.759 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:45d2fc81:::benchmark_data_smithi039_468595_object142:head# 2024-09-16T17:24:09.759 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.792 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4609546b:::benchmark_data_smithi039_468595_object79:head# 2024-09-16T17:24:09.793 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.826 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4720d590:::benchmark_data_smithi039_468595_object107:head# 2024-09-16T17:24:09.826 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.859 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:496e90ea:::benchmark_data_smithi039_468595_object110:head# 2024-09-16T17:24:09.859 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.893 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4a17d722:::benchmark_data_smithi039_468595_object202:head# 2024-09-16T17:24:09.893 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.926 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4a1b0efc:::benchmark_data_smithi039_468595_object32:head# 2024-09-16T17:24:09.926 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.960 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4a4bfa19:::benchmark_data_smithi039_468595_object22:head# 2024-09-16T17:24:09.960 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:09.993 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4b08ba44:::benchmark_data_smithi039_468595_object26:head# 2024-09-16T17:24:09.993 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.027 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4badec40:::benchmark_data_smithi039_468595_object85:head# 2024-09-16T17:24:10.027 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.068 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4bb4463a:::benchmark_data_smithi039_468595_object225:head# 2024-09-16T17:24:10.069 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.135 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4c6af759:::benchmark_data_smithi039_468595_object82:head# 2024-09-16T17:24:10.135 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.169 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4cec54db:::benchmark_data_smithi039_468595_object134:head# 2024-09-16T17:24:10.169 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.202 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4d5c4ae5:::benchmark_data_smithi039_468595_object253:head# 2024-09-16T17:24:10.203 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.236 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4d5f10b6:::benchmark_data_smithi039_468595_object328:head# 2024-09-16T17:24:10.236 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.269 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4d96ad59:::benchmark_data_smithi039_468595_object171:head# 2024-09-16T17:24:10.269 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.303 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4f1bbd9e:::benchmark_data_smithi039_468595_object68:head# 2024-09-16T17:24:10.303 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.343 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4ffc3aa2:::benchmark_data_smithi039_468595_object105:head# 2024-09-16T17:24:10.344 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.420 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:50903a7c:::benchmark_data_smithi039_468595_object233:head# 2024-09-16T17:24:10.420 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.496 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:50a0457c:::benchmark_data_smithi039_468595_object209:head# 2024-09-16T17:24:10.496 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.537 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:52207ac9:::benchmark_data_smithi039_468595_object124:head# 2024-09-16T17:24:10.537 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.579 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:52337313:::benchmark_data_smithi039_468595_object217:head# 2024-09-16T17:24:10.579 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.621 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53e09036:::benchmark_data_smithi039_468595_object144:head# 2024-09-16T17:24:10.621 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.663 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5463a2e4:::benchmark_data_smithi039_468595_object307:head# 2024-09-16T17:24:10.663 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.705 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:54ba4047:::benchmark_data_smithi039_468595_object238:head# 2024-09-16T17:24:10.705 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.746 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:54e01bd5:::benchmark_data_smithi039_468595_object69:head# 2024-09-16T17:24:10.746 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.788 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:564adc8d:::benchmark_data_smithi039_468595_object197:head# 2024-09-16T17:24:10.788 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.830 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5687d21e:::benchmark_data_smithi039_468595_object91:head# 2024-09-16T17:24:10.830 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.872 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5707ca11:::benchmark_data_smithi039_468595_object11:head# 2024-09-16T17:24:10.872 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.914 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:57a103a8:::benchmark_data_smithi039_468595_object99:head# 2024-09-16T17:24:10.914 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.955 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:57e99a09:::benchmark_data_smithi039_468595_object199:head# 2024-09-16T17:24:10.955 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:10.997 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:585a6e4b:::benchmark_data_smithi039_468595_object104:head# 2024-09-16T17:24:10.997 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.054 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:59592b00:::benchmark_data_smithi039_468595_object168:head# 2024-09-16T17:24:11.054 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.081 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:598e986c:::benchmark_data_smithi039_468595_object237:head# 2024-09-16T17:24:11.081 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.114 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:59a63be9:::benchmark_data_smithi039_468595_object294:head# 2024-09-16T17:24:11.114 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.148 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5a6259af:::benchmark_data_smithi039_468595_object147:head# 2024-09-16T17:24:11.148 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.181 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ab586e6:::benchmark_data_smithi039_468595_object28:head# 2024-09-16T17:24:11.181 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.214 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5bf7b5cb:::benchmark_data_smithi039_468595_object207:head# 2024-09-16T17:24:11.215 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.248 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5c5994f6:::benchmark_data_smithi039_468595_object157:head# 2024-09-16T17:24:11.248 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.281 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5cd70c81:::benchmark_data_smithi039_468595_object162:head# 2024-09-16T17:24:11.282 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.323 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5d78aa96:::benchmark_data_smithi039_468595_object302:head# 2024-09-16T17:24:11.323 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.357 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5d7f70d7:::benchmark_data_smithi039_468595_object60:head# 2024-09-16T17:24:11.357 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.390 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5df833aa:::benchmark_data_smithi039_468595_object284:head# 2024-09-16T17:24:11.391 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.424 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ea7bc08:::benchmark_data_smithi039_468595_object92:head# 2024-09-16T17:24:11.424 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.457 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ee3e58e:::benchmark_data_smithi039_468595_object223:head# 2024-09-16T17:24:11.457 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.491 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:62a8fa88:::benchmark_data_smithi039_468595_object56:head# 2024-09-16T17:24:11.491 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.524 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:62ee6725:::benchmark_data_smithi039_468595_object44:head# 2024-09-16T17:24:11.524 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.558 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:64ab38c2:::benchmark_data_smithi039_468595_object89:head# 2024-09-16T17:24:11.558 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.591 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6649a7f0:::benchmark_data_smithi039_468595_object240:head# 2024-09-16T17:24:11.591 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.624 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:66740cbf:::benchmark_data_smithi039_468595_object215:head# 2024-09-16T17:24:11.624 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.666 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6d6ec100:::benchmark_data_smithi039_468595_object324:head# 2024-09-16T17:24:11.666 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.700 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6d7e0208:::benchmark_data_smithi039_468595_object143:head# 2024-09-16T17:24:11.700 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.733 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6dc530d1:::benchmark_data_smithi039_468595_object274:head# 2024-09-16T17:24:11.733 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.767 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6e432b0c:::benchmark_data_smithi039_468595_object333:head# 2024-09-16T17:24:11.767 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.800 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6ea23d44:::benchmark_data_smithi039_468595_object259:head# 2024-09-16T17:24:11.800 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6edc8698:::benchmark_data_smithi039_468595_object299:head# 2024-09-16T17:24:11.834 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6ef8850a:::benchmark_data_smithi039_468595_object112:head# 2024-09-16T17:24:11.867 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.900 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6fcac29e:::benchmark_data_smithi039_468595_object46:head# 2024-09-16T17:24:11.901 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.934 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:703dce8c:::benchmark_data_smithi039_468595_object303:head# 2024-09-16T17:24:11.934 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:11.968 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:714274ce:::benchmark_data_smithi039_468595_object98:head# 2024-09-16T17:24:11.968 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.001 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:721fa733:::benchmark_data_smithi039_468595_object332:head# 2024-09-16T17:24:12.001 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.034 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:722f4cb7:::benchmark_data_smithi039_468595_object219:head# 2024-09-16T17:24:12.035 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.076 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:72ea8d9f:::benchmark_data_smithi039_468595_object87:head# 2024-09-16T17:24:12.077 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.151 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:73f7fc61:::benchmark_data_smithi039_468595_object4:head# 2024-09-16T17:24:12.152 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.185 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:73fcfe0e:::benchmark_data_smithi039_468595_object205:head# 2024-09-16T17:24:12.185 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.219 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:754c286d:::benchmark_data_smithi039_468595_object41:head# 2024-09-16T17:24:12.219 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.252 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:768427e5:::benchmark_data_smithi039_468595_object260:head# 2024-09-16T17:24:12.252 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.286 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7685c881:::benchmark_data_smithi039_468595_object123:head# 2024-09-16T17:24:12.286 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.319 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:77e63c42:::benchmark_data_smithi039_468595_object231:head# 2024-09-16T17:24:12.319 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.413 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:790cd8e2:::benchmark_data_smithi039_468595_object19:head# 2024-09-16T17:24:12.413 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.487 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7baf820d:::benchmark_data_smithi039_468595_object159:head# 2024-09-16T17:24:12.487 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.520 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7cafc3a6:::benchmark_data_smithi039_468595_object38:head# 2024-09-16T17:24:12.520 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.554 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7f14319c:::benchmark_data_smithi039_468595_object245:head# 2024-09-16T17:24:12.554 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.587 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:80918c14:::benchmark_data_smithi039_468595_object16:head# 2024-09-16T17:24:12.587 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.621 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:80d9f722:::benchmark_data_smithi039_468595_object106:head# 2024-09-16T17:24:12.621 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.654 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:80fce290:::benchmark_data_smithi039_468595_object184:head# 2024-09-16T17:24:12.654 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.687 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81f53868:::benchmark_data_smithi039_468595_object298:head# 2024-09-16T17:24:12.688 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.729 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:844f9018:::benchmark_data_smithi039_468595_object262:head# 2024-09-16T17:24:12.729 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.763 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:84bb7162:::benchmark_data_smithi039_468595_object311:head# 2024-09-16T17:24:12.763 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.796 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:850b1a92:::benchmark_data_smithi039_468595_object138:head# 2024-09-16T17:24:12.796 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.830 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8569d272:::benchmark_data_smithi039_468595_object241:head# 2024-09-16T17:24:12.830 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.863 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:863caf4a:::benchmark_data_smithi039_468595_object330:head# 2024-09-16T17:24:12.863 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.905 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:869f7a6a:::benchmark_data_smithi039_468595_object265:head# 2024-09-16T17:24:12.905 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:12.963 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8701fee5:::benchmark_data_smithi039_468595_object135:head# 2024-09-16T17:24:12.964 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.005 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:880a2a14:::benchmark_data_smithi039_468595_object52:head# 2024-09-16T17:24:13.005 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.047 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:887b6945:::benchmark_data_smithi039_468595_object141:head# 2024-09-16T17:24:13.047 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.089 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:88fcacfa:::benchmark_data_smithi039_468595_object25:head# 2024-09-16T17:24:13.089 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.131 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:891807e1:::benchmark_data_smithi039_468595_object114:head# 2024-09-16T17:24:13.131 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.172 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:89d1091d:::benchmark_data_smithi039_468595_object0:head# 2024-09-16T17:24:13.173 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.214 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8b43ece7:::benchmark_data_smithi039_468595_object81:head# 2024-09-16T17:24:13.214 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.256 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8dace154:::benchmark_data_smithi039_468595_object173:head# 2024-09-16T17:24:13.256 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.298 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8de5b1c0:::benchmark_data_smithi039_468595_object163:head# 2024-09-16T17:24:13.298 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.340 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8e6fcc3f:::benchmark_data_smithi039_468595_object29:head# 2024-09-16T17:24:13.340 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.393 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8f0ce81c:::benchmark_data_smithi039_468595_object94:head# 2024-09-16T17:24:13.393 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.483 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8fcb7113:::benchmark_data_smithi039_468595_object266:head# 2024-09-16T17:24:13.483 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.566 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9045a82d:::benchmark_data_smithi039_468595_object43:head# 2024-09-16T17:24:13.566 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.626 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:91f2e0f9:::benchmark_data_smithi039_468595_object64:head# 2024-09-16T17:24:13.627 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.723 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9237253d:::benchmark_data_smithi039_468595_object329:head# 2024-09-16T17:24:13.723 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.775 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:926c2408:::benchmark_data_smithi039_468595_object301:head# 2024-09-16T17:24:13.775 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.809 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92897a94:::benchmark_data_smithi039_468595_object156:head# 2024-09-16T17:24:13.809 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.842 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92c8bc05:::benchmark_data_smithi039_468595_object97:head# 2024-09-16T17:24:13.842 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.876 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:93531de3:::benchmark_data_smithi039_468595_object78:head# 2024-09-16T17:24:13.876 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.909 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:97a77275:::benchmark_data_smithi039_468595_object242:head# 2024-09-16T17:24:13.909 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.943 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:97b3ce3c:::benchmark_data_smithi039_468595_object305:head# 2024-09-16T17:24:13.943 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:13.985 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:981dc61a:::benchmark_data_smithi039_468595_object195:head# 2024-09-16T17:24:13.985 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.018 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:98aa50c6:::benchmark_data_smithi039_468595_object276:head# 2024-09-16T17:24:14.018 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.052 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:99e59096:::benchmark_data_smithi039_468595_object161:head# 2024-09-16T17:24:14.052 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.102 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:99ed60a7:::benchmark_data_smithi039_468595_object17:head# 2024-09-16T17:24:14.102 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.160 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9a14b9a4:::benchmark_data_smithi039_468595_object50:head# 2024-09-16T17:24:14.161 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.194 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9b038352:::benchmark_data_smithi039_468595_object224:head# 2024-09-16T17:24:14.194 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.228 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9b1f346f:::benchmark_data_smithi039_468595_object30:head# 2024-09-16T17:24:14.228 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.261 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9b63c8c4:::benchmark_data_smithi039_468595_object318:head# 2024-09-16T17:24:14.261 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.294 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9bc8111e:::benchmark_data_smithi039_468595_object212:head# 2024-09-16T17:24:14.294 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.334 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9c275498:::benchmark_data_smithi039_468595_object297:head# 2024-09-16T17:24:14.334 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.422 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9c2f4e7a:::benchmark_data_smithi039_468595_object58:head# 2024-09-16T17:24:14.423 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.487 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9cfeb30a:::benchmark_data_smithi039_468595_object316:head# 2024-09-16T17:24:14.487 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.521 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9e572ace:::benchmark_data_smithi039_468595_object125:head# 2024-09-16T17:24:14.521 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.554 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9fe9f39f:::benchmark_data_smithi039_468595_object335:head# 2024-09-16T17:24:14.554 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.587 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a1aa6f8e:::benchmark_data_smithi039_468595_object291:head# 2024-09-16T17:24:14.588 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.621 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a1fbc247:::benchmark_data_smithi039_468595_object186:head# 2024-09-16T17:24:14.621 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.655 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a217378b:::benchmark_data_smithi039_468595_object228:head# 2024-09-16T17:24:14.655 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.688 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a25c34de:::benchmark_data_smithi039_468595_object182:head# 2024-09-16T17:24:14.688 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.721 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a376bc42:::benchmark_data_smithi039_468595_object309:head# 2024-09-16T17:24:14.722 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.755 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a4c17a67:::benchmark_data_smithi039_468595_object187:head# 2024-09-16T17:24:14.755 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.788 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a7a04742:::benchmark_data_smithi039_468595_object317:head# 2024-09-16T17:24:14.789 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.822 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a8eb188c:::benchmark_data_smithi039_468595_object153:head# 2024-09-16T17:24:14.822 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.855 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a936fd24:::benchmark_data_smithi039_468595_object76:head# 2024-09-16T17:24:14.855 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.889 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ac3170c8:::benchmark_data_smithi039_468595_object146:head# 2024-09-16T17:24:14.889 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.922 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ace79a8c:::benchmark_data_smithi039_468595_object48:head# 2024-09-16T17:24:14.922 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.956 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ad369937:::benchmark_data_smithi039_468595_object188:head# 2024-09-16T17:24:14.956 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:14.989 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aeca547f:::benchmark_data_smithi039_468595_object200:head# 2024-09-16T17:24:14.989 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.023 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aed81ba6:::benchmark_data_smithi039_468595_object15:head# 2024-09-16T17:24:15.023 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.065 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b0201ed8:::benchmark_data_smithi039_468595_object211:head# 2024-09-16T17:24:15.065 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.106 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b0f1c022:::benchmark_data_smithi039_468595_object252:head# 2024-09-16T17:24:15.106 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.148 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b1e5f2ad:::benchmark_data_smithi039_468595_object18:head# 2024-09-16T17:24:15.148 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.190 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b2d0bf87:::benchmark_data_smithi039_468595_object120:head# 2024-09-16T17:24:15.190 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.232 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b344ddc1:::benchmark_data_smithi039_468595_object36:head# 2024-09-16T17:24:15.232 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.273 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b357dd0c:::benchmark_data_smithi039_468595_object39:head# 2024-09-16T17:24:15.273 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.315 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b46a710e:::benchmark_data_smithi039_468595_object164:head# 2024-09-16T17:24:15.315 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.357 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b57c02fe:::benchmark_data_smithi039_468595_object285:head# 2024-09-16T17:24:15.357 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.424 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b5878bb3:::benchmark_data_smithi039_468595_object254:head# 2024-09-16T17:24:15.424 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.457 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b601fe0a:::benchmark_data_smithi039_468595_object57:head# 2024-09-16T17:24:15.457 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.491 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b67d6553:::benchmark_data_smithi039_468595_object100:head# 2024-09-16T17:24:15.491 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.524 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b722fd33:::benchmark_data_smithi039_468595_object282:head# 2024-09-16T17:24:15.524 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.558 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b7a78853:::benchmark_data_smithi039_468595_object66:head# 2024-09-16T17:24:15.558 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.591 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b8163b8e:::benchmark_data_smithi039_468595_object176:head# 2024-09-16T17:24:15.591 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.633 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-16T17:24:15.633 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.675 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b885a7d2:::benchmark_data_smithi039_468595_object166:head# 2024-09-16T17:24:15.675 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.717 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b8d3729a:::benchmark_data_smithi039_468595_object268:head# 2024-09-16T17:24:15.717 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.750 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bbafc8a3:::benchmark_data_smithi039_468595_object111:head# 2024-09-16T17:24:15.750 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.784 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bbd3b87a:::benchmark_data_smithi039_468595_object145:head# 2024-09-16T17:24:15.784 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.817 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bc22f79f:::benchmark_data_smithi039_468595_object189:head# 2024-09-16T17:24:15.817 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.851 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bd7ee7ea:::benchmark_data_smithi039_468595_object235:head# 2024-09-16T17:24:15.851 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.884 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bd9dabc0:::benchmark_data_smithi039_468595_object220:head# 2024-09-16T17:24:15.884 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.917 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bdd0883c:::benchmark_data_smithi039_468595_object3:head# 2024-09-16T17:24:15.917 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.951 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:beb7571d:::benchmark_data_smithi039_468595_object49:head# 2024-09-16T17:24:15.951 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:15.984 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c0a2001b:::benchmark_data_smithi039_468595_object244:head# 2024-09-16T17:24:15.985 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.018 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c0e305af:::benchmark_data_smithi039_468595_object288:head# 2024-09-16T17:24:16.018 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.051 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c1a6dd82:::benchmark_data_smithi039_468595_object132:head# 2024-09-16T17:24:16.051 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.118 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c3460b41:::benchmark_data_smithi039_468595_object175:head# 2024-09-16T17:24:16.118 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.169 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c3879720:::benchmark_data_smithi039_468595_object47:head# 2024-09-16T17:24:16.169 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.202 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c57a1706:::benchmark_data_smithi039_468595_object179:head# 2024-09-16T17:24:16.202 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.235 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c58f9a8e:::benchmark_data_smithi039_468595_object257:head# 2024-09-16T17:24:16.236 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.269 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c8eedfc1:::benchmark_data_smithi039_468595_object174:head# 2024-09-16T17:24:16.269 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.302 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c99442a0:::benchmark_data_smithi039_468595_object321:head# 2024-09-16T17:24:16.303 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.342 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ca33a770:::benchmark_data_smithi039_468595_object210:head# 2024-09-16T17:24:16.342 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.430 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cb34b9ee:::benchmark_data_smithi039_468595_object55:head# 2024-09-16T17:24:16.430 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.495 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cb7004f2:::benchmark_data_smithi039_468595_object167:head# 2024-09-16T17:24:16.495 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.545 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ce600566:::benchmark_data_smithi039_468595_object126:head# 2024-09-16T17:24:16.546 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.579 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cf33d8ae:::benchmark_data_smithi039_468595_object62:head# 2024-09-16T17:24:16.579 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.613 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cf3bb6d3:::benchmark_data_smithi039_468595_object118:head# 2024-09-16T17:24:16.613 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.646 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cfbe5e78:::benchmark_data_smithi039_468595_object103:head# 2024-09-16T17:24:16.646 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.679 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d07a3480:::benchmark_data_smithi039_468595_object165:head# 2024-09-16T17:24:16.680 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.713 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d0bfbe29:::benchmark_data_smithi039_468595_object12:head# 2024-09-16T17:24:16.713 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.746 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d0f9d51d:::benchmark_data_smithi039_468595_object194:head# 2024-09-16T17:24:16.746 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.780 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d289528f:::benchmark_data_smithi039_468595_object264:head# 2024-09-16T17:24:16.780 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.813 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d30255de:::benchmark_data_smithi039_468595_object226:head# 2024-09-16T17:24:16.813 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.846 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d3184e15:::benchmark_data_smithi039_468595_object73:head# 2024-09-16T17:24:16.847 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.880 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d351bdaf:::benchmark_data_smithi039_468595_object249:head# 2024-09-16T17:24:16.880 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.913 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d47ba051:::benchmark_data_smithi039_468595_object121:head# 2024-09-16T17:24:16.914 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.947 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d4936f9d:::benchmark_data_smithi039_468595_object115:head# 2024-09-16T17:24:16.947 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:16.980 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d4bd7d38:::benchmark_data_smithi039_468595_object278:head# 2024-09-16T17:24:16.980 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.014 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d4be707a:::benchmark_data_smithi039_468595_object273:head# 2024-09-16T17:24:17.014 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.047 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d52ca634:::benchmark_data_smithi039_468595_object151:head# 2024-09-16T17:24:17.048 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.081 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d7dfb1a9:::benchmark_data_smithi039_468595_object150:head# 2024-09-16T17:24:17.081 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.114 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d7eb6c60:::benchmark_data_smithi039_468595_object88:head# 2024-09-16T17:24:17.114 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.148 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d8a669c3:::benchmark_data_smithi039_468595_object191:head# 2024-09-16T17:24:17.148 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.181 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dbcf17ab:::benchmark_data_smithi039_468595_object322:head# 2024-09-16T17:24:17.181 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.214 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dbf09c2f:::benchmark_data_smithi039_468595_object221:head# 2024-09-16T17:24:17.215 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.248 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dc452c3b:::benchmark_data_smithi039_468595_object13:head# 2024-09-16T17:24:17.248 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.290 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dc809645:::benchmark_data_smithi039_468595_object27:head# 2024-09-16T17:24:17.290 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.323 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dcf39b6c:::benchmark_data_smithi039_468595_object310:head# 2024-09-16T17:24:17.323 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.357 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ddb3c124:::benchmark_data_smithi039_468595_object154:head# 2024-09-16T17:24:17.357 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.390 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:de15f6ae:::benchmark_data_smithi039_468595_object327:head# 2024-09-16T17:24:17.390 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.432 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:de2003b5:::benchmark_data_smithi039_468595_object71:head# 2024-09-16T17:24:17.432 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.465 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:df17aa0c:::benchmark_data_smithi039_468595_object53:head# 2024-09-16T17:24:17.465 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.507 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:df436e46:::benchmark_data_smithi039_468595_object277:head# 2024-09-16T17:24:17.507 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e0539c89:::benchmark_data_smithi039_468595_object246:head# 2024-09-16T17:24:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.591 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e09fbc15:::benchmark_data_smithi039_468595_object181:head# 2024-09-16T17:24:17.591 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.632 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e1f04201:::benchmark_data_smithi039_468595_object80:head# 2024-09-16T17:24:17.633 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.683 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e3ab9a7b:::benchmark_data_smithi039_468595_object315:head# 2024-09-16T17:24:17.683 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.716 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e3d55e4c:::benchmark_data_smithi039_468595_object255:head# 2024-09-16T17:24:17.717 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.750 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e3ee6c37:::benchmark_data_smithi039_468595_object287:head# 2024-09-16T17:24:17.750 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.783 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e4fd5aea:::benchmark_data_smithi039_468595_object331:head# 2024-09-16T17:24:17.783 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.817 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e595dc31:::benchmark_data_smithi039_468595_object193:head# 2024-09-16T17:24:17.817 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.850 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e6d657ff:::benchmark_data_smithi039_468595_object313:head# 2024-09-16T17:24:17.850 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.884 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e7288009:::benchmark_data_smithi039_468595_object160:head# 2024-09-16T17:24:17.884 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.917 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e781cf57:::benchmark_data_smithi039_468595_object272:head# 2024-09-16T17:24:17.917 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.951 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e7a5c1a0:::benchmark_data_smithi039_468595_object137:head# 2024-09-16T17:24:17.951 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:17.984 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e7ae1cf0:::benchmark_data_smithi039_468595_object203:head# 2024-09-16T17:24:17.984 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.017 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e8c79fcc:::benchmark_data_smithi039_468595_object65:head# 2024-09-16T17:24:18.018 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.051 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e9c1a181:::benchmark_data_smithi039_468595_object40:head# 2024-09-16T17:24:18.051 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.118 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ea98c336:::benchmark_data_smithi039_468595_object77:head# 2024-09-16T17:24:18.118 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.168 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:eb115269:::benchmark_data_smithi039_468595_object319:head# 2024-09-16T17:24:18.168 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.202 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:eb1e19f4:::benchmark_data_smithi039_468595_object177:head# 2024-09-16T17:24:18.202 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.243 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:eb9c8a80:::benchmark_data_smithi039_468595_object300:head# 2024-09-16T17:24:18.243 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.277 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ebbadbe5:::benchmark_data_smithi039_468595_object158:head# 2024-09-16T17:24:18.277 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.311 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ec76a88e:::benchmark_data_smithi039_468595_object23:head# 2024-09-16T17:24:18.311 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.386 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:eca7d2be:::benchmark_data_smithi039_468595_object169:head# 2024-09-16T17:24:18.386 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.445 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ed0424ec:::benchmark_data_smithi039_468595_object289:head# 2024-09-16T17:24:18.445 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.662 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:edfb9ce3:::benchmark_data_smithi039_468595_object216:head# 2024-09-16T17:24:18.662 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.695 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ee55e927:::benchmark_data_smithi039_468595_object155:head# 2024-09-16T17:24:18.695 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.729 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ef41dc71:::benchmark_data_smithi039_468595_object139:head# 2024-09-16T17:24:18.729 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.762 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ef47a94f:::benchmark_data_smithi039_468595_object130:head# 2024-09-16T17:24:18.762 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.796 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:efa9c4c9:::benchmark_data_smithi039_468595_object206:head# 2024-09-16T17:24:18.796 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.829 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f080017e:::benchmark_data_smithi039_468595_object86:head# 2024-09-16T17:24:18.829 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.863 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f13659e9:::benchmark_data_smithi039_468595_object304:head# 2024-09-16T17:24:18.863 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.896 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f3269939:::benchmark_data_smithi039_468595_object83:head# 2024-09-16T17:24:18.896 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.930 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f4549a74:::benchmark_data_smithi039_468595_object314:head# 2024-09-16T17:24:18.930 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.963 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f582b90f:::benchmark_data_smithi039_468595_object198:head# 2024-09-16T17:24:18.963 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:18.997 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f62d4d3c:::benchmark_data_smithi039_468595_object227:head# 2024-09-16T17:24:18.997 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.030 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f750af9a:::benchmark_data_smithi039_468595_object35:head# 2024-09-16T17:24:19.030 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.105 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f7fa6095:::benchmark_data_smithi039_468595_object117:head# 2024-09-16T17:24:19.105 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.139 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f801d8a3:::benchmark_data_smithi039_468595_object267:head# 2024-09-16T17:24:19.139 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.172 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f8ed550b:::benchmark_data_smithi039_468595_object149:head# 2024-09-16T17:24:19.173 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.206 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f910a5fe:::benchmark_data_smithi039_468595_object192:head# 2024-09-16T17:24:19.206 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.248 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f910f5db:::benchmark_data_smithi039_468595_object84:head# 2024-09-16T17:24:19.248 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.281 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f94d74ec:::benchmark_data_smithi039_468595_object31:head# 2024-09-16T17:24:19.281 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.315 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f9debaaf:::benchmark_data_smithi039_468595_object320:head# 2024-09-16T17:24:19.315 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.348 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fa5fc20d:::benchmark_data_smithi039_468595_object275:head# 2024-09-16T17:24:19.348 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.381 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fb817c6e:::benchmark_data_smithi039_468595_object119:head# 2024-09-16T17:24:19.382 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.415 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fbc18c31:::benchmark_data_smithi039_468595_object8:head# 2024-09-16T17:24:19.415 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.448 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fc0d8571:::benchmark_data_smithi039_468595_object93:head# 2024-09-16T17:24:19.449 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.482 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fc0f46c2:::benchmark_data_smithi039_468595_object190:head# 2024-09-16T17:24:19.482 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.515 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fc0f9e7a:::benchmark_data_smithi039_468595_object312:head# 2024-09-16T17:24:19.516 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.549 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fe77dedf:::benchmark_data_smithi039_468595_object296:head# 2024-09-16T17:24:19.549 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.582 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:feca5f29:::benchmark_data_smithi039_468595_object33:head# 2024-09-16T17:24:19.582 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.616 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fef32071:::benchmark_data_smithi039_468595_object42:head# 2024-09-16T17:24:19.616 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.649 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ff290914:::benchmark_data_smithi039_468595_object133:head# 2024-09-16T17:24:19.649 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-16T17:24:19.683 INFO:tasks.workunit.client.0.smithi039.stdout:write_pg epoch 9 info 1.0( v 9'338 (0'0,9'338] local-lis/les=7/8 n=338 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-16T17:24:19.758 INFO:tasks.workunit.client.0.smithi039.stdout:Import successful 2024-09-16T17:24:20.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:193: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-09-16T17:24:20.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-16T17:24:20.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T17:24:20.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T17:24:20.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T17:24:20.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-16T17:24:20.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-16T17:24:20.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:24:20.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:24:20.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:24:20.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-16T17:24:20.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-16T17:24:20.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T17:24:20.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T17:24:20.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-16T17:24:20.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T17:24:20.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:24:20.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:24:20.303 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:24:20.303 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:24:20.303 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:24:20.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:24:20.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' 2024-09-16T17:24:20.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:24:20.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-16T17:24:20.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-16T17:24:20.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:24:20.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:24:20.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:24:20.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:24:20.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T17:24:20.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T17:24:20.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-16T17:24:20.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T17:24:20.305 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:24:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7cc96025-8033-4ad0-8155-ad971471fe5c --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.444977/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:24:20.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-16T17:24:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T17:24:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T17:24:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T17:24:20.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T17:24:20.340 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:20.338+0000 7fa12dd0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:24:20.351 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:20.350+0000 7fa12dd0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:24:20.353 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:20.350+0000 7fa12dd0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:24:20.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T17:24:20.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:24:20.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:24:20.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:24:20.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:24:20.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:20.784 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:24:20.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:24:20.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:20.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:20.901 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:20.898+0000 7fa12dd0c7c0 -1 Falling back to public interface 2024-09-16T17:24:21.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:22.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:22.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:22.206 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:24:22.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:24:22.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:22.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:22.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:23.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:23.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:23.658 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:24:23.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:24:23.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:23.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:24.003 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:23.998+0000 7fa12dd0c7c0 -1 osd.0 18 log_to_monitors true 2024-09-16T17:24:24.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:25.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:25.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:25.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:24:25.096 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:24:25.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:25.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:25.631 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [12,18) [v2:127.0.0.1:6802/2279515841,v1:127.0.0.1:6803/2279515841] [v2:127.0.0.1:6804/2279515841,v1:127.0.0.1:6805/2279515841] exists,up c1945d7e-0f4d-49d8-91bb-17a1555e217a 2024-09-16T17:24:25.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:195: TEST_import_after_split: wait_for_clean 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:24:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:24:25.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:24:25.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:24:25.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:24:25.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:24:26.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T17:24:26.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:24:26.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:24:26.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:24:26.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=90194313219 2024-09-16T17:24:26.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 90194313219 2024-09-16T17:24:26.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-90194313219' 2024-09-16T17:24:26.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:24:26.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-90194313219 2024-09-16T17:24:26.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:24:26.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:24:26.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-90194313219 2024-09-16T17:24:26.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:24:26.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=90194313219 2024-09-16T17:24:26.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 90194313219' 2024-09-16T17:24:26.368 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 90194313219 2024-09-16T17:24:26.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:24:26.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 90194313219 2024-09-16T17:24:26.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:24:27.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:24:27.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:24:28.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 90194313219 2024-09-16T17:24:28.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:24:29.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-16T17:24:29.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:24:29.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 90194313219 -lt 90194313219 2024-09-16T17:24:29.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:24:29.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:24:29.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-16T17:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:24:30.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:24:30.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:24:30.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-16T17:24:30.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:24:30.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:24:30.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:24:31.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-16T17:24:31.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:24:31.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:24:31.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-09-16T17:24:31.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-16T17:24:31.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:24:31.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-16T17:24:31.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:24:31.212 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:24:31.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:24:31.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:24:31.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:24:31.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:24:31.319 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:24:31.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:24:31.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:24:31.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:24:31.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:24:31.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:24:31.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:24:31.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:24:31.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:24:31.323 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:24:31.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:24:31.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:24:31.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-16T17:24:31.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:24:31.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:24:31.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:24:31.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.444977 2024-09-16T17:24:31.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:24:31.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:24:31.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T17:24:31.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/pg-split-merge 0 2024-09-16T17:24:31.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-16T17:24:31.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T17:24:31.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-16T17:24:31.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:24:31.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:24:31.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:24:31.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:24:31.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:24:31.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:24:31.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:24:31.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:24:31.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:24:31.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:24:31.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:24:31.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:24:31.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:24:31.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:24:31.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:24:31.341 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:24:31.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:24:31.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T17:24:31.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-16T17:24:31.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:24:31.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:24:31.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444977 2024-09-16T17:24:31.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.444977 2024-09-16T17:24:31.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:24:31.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:24:31.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T17:24:31.345 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T17:24:31.346 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T17:24:31.399 INFO:tasks.workunit:Running workunit osd/repeer-on-acting-back.sh... 2024-09-16T17:24:31.400 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/repeer-on-acting-back.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh 2024-09-16T17:24:31.452 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T17:24:31.456 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/repeer-on-acting-back 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:22: run: local dir=td/repeer-on-acting-back 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:23: run: shift 2024-09-16T17:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: export poolname=test 2024-09-16T17:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: poolname=test 2024-09-16T17:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: export testobjects=100 2024-09-16T17:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: testobjects=100 2024-09-16T17:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: export loglen=12 2024-09-16T17:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: loglen=12 2024-09-16T17:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: expr 12 / 2 2024-09-16T17:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: export trim=6 2024-09-16T17:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: trim=6 2024-09-16T17:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: export CEPH_MON=127.0.0.1:7115 2024-09-16T17:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: CEPH_MON=127.0.0.1:7115 2024-09-16T17:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:30: run: export CEPH_ARGS 2024-09-16T17:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: uuidgen 2024-09-16T17:24:31.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: CEPH_ARGS+='--fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none ' 2024-09-16T17:24:31.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-16T17:24:31.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:34: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-16T17:24:31.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:36: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T17:24:31.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: set 2024-09-16T17:24:31.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: local funcs=TEST_repeer_on_down_acting_member_coming_back 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:39: run: for func in $funcs 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:40: run: setup td/repeer-on-acting-back 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repeer-on-acting-back 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repeer-on-acting-back 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:24:31.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:24:31.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:24:31.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:24:31.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:24:31.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:24:31.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:24:31.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:24:31.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:24:31.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:24:31.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:24:31.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:24:31.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:24:31.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:24:31.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:24:31.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-09-16T17:24:31.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:24:31.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:24:31.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:24:31.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.472214 2024-09-16T17:24:31.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:24:31.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:24:31.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repeer-on-acting-back 2024-09-16T17:24:31.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:24:31.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:24:31.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:24:31.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.472214 2024-09-16T17:24:31.476 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:24:31.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T17:24:31.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T17:24:31.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:24:31.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repeer-on-acting-back 1' TERM HUP INT 2024-09-16T17:24:31.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:41: run: TEST_repeer_on_down_acting_member_coming_back td/repeer-on-acting-back 2024-09-16T17:24:31.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:48: TEST_repeer_on_down_acting_member_coming_back: local dir=td/repeer-on-acting-back 2024-09-16T17:24:31.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:49: TEST_repeer_on_down_acting_member_coming_back: local dummyfile=/etc/fstab 2024-09-16T17:24:31.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:51: TEST_repeer_on_down_acting_member_coming_back: local num_osds=6 2024-09-16T17:24:31.477 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: expr 6 - 1 2024-09-16T17:24:31.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: seq 0 5 2024-09-16T17:24:31.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: local 'osds=0 2024-09-16T17:24:31.478 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:24:31.478 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:24:31.479 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T17:24:31.479 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T17:24:31.479 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T17:24:31.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:53: TEST_repeer_on_down_acting_member_coming_back: run_mon td/repeer-on-acting-back a 2024-09-16T17:24:31.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repeer-on-acting-back 2024-09-16T17:24:31.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:24:31.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:24:31.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:24:31.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repeer-on-acting-back/a 2024-09-16T17:24:31.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repeer-on-acting-back/a --run-dir=td/repeer-on-acting-back 2024-09-16T17:24:31.882 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:24:31.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:24:31.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:24:31.883 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:24:31.883 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:24:31.883 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:24:31.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:24:31.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repeer-on-acting-back/a '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --mon-cluster-log-file=td/repeer-on-acting-back/log --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T17:24:32.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:24:32.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:24:32.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:24:32.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:24:32.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:24:32.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:24:32.096 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:24:32.096 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:24:32.096 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:24:32.097 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:24:32.097 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:24:32.097 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:24:32.097 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.472214/ceph-mon.a.asok 2024-09-16T17:24:32.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:24:32.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.472214/ceph-mon.a.asok config get fsid 2024-09-16T17:24:32.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:24:32.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:24:32.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:24:32.213 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:24:32.214 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:24:32.214 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:24:32.214 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:24:32.214 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:24:32.214 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:24:32.214 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:24:32.214 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:24:32.215 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.472214/ceph-mon.a.asok 2024-09-16T17:24:32.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:24:32.215 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.472214/ceph-mon.a.asok config get mon_host 2024-09-16T17:24:32.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:54: TEST_repeer_on_down_acting_member_coming_back: run_mgr td/repeer-on-acting-back x 2024-09-16T17:24:32.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repeer-on-acting-back 2024-09-16T17:24:32.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:24:32.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:24:32.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:24:32.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repeer-on-acting-back/x 2024-09-16T17:24:32.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:24:33.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:24:33.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:24:33.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:24:33.282 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:24:33.283 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:24:33.283 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:24:33.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:24:33.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:24:33.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repeer-on-acting-back/x '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:24:33.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 0 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/0 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/0' 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/0/journal' 2024-09-16T17:24:33.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:24:33.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-16T17:24:33.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-16T17:24:33.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:24:33.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:24:33.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:24:33.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:24:33.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:24:33.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:24:33.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/0 2024-09-16T17:24:33.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:24:33.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=074c2305-9519-4c47-89d0-96827be4a349 2024-09-16T17:24:33.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 074c2305-9519-4c47-89d0-96827be4a349' 2024-09-16T17:24:33.364 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 074c2305-9519-4c47-89d0-96827be4a349 2024-09-16T17:24:33.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:24:33.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRaehmDWzyFhAA3N50X+ZC4r1KUUvXOPYO9w== 2024-09-16T17:24:33.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRaehmDWzyFhAA3N50X+ZC4r1KUUvXOPYO9w=="}' 2024-09-16T17:24:33.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 074c2305-9519-4c47-89d0-96827be4a349 -i td/repeer-on-acting-back/0/new.json 2024-09-16T17:24:33.722 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:24:33.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/0/new.json 2024-09-16T17:24:33.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRaehmDWzyFhAA3N50X+ZC4r1KUUvXOPYO9w== --osd-uuid 074c2305-9519-4c47-89d0-96827be4a349 2024-09-16T17:24:33.772 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:33.770+0000 7f127f1397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:24:33.782 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:33.782+0000 7f127f1397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:24:33.784 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:33.782+0000 7f127f1397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:24:33.784 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:33.782+0000 7f127f1397c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-09-16T17:24:33.784 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:33.782+0000 7f127f1397c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-09-16T17:24:33.784 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:33.782+0000 7f127f1397c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-09-16T17:24:33.823 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:33.822+0000 7f127f1397c0 -1 bluestore(td/repeer-on-acting-back/0) _read_fsid unparsable uuid 2024-09-16T17:24:40.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/0/keyring 2024-09-16T17:24:40.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:24:40.201 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:24:40.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:24:40.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:24:40.759 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:24:40.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:24:40.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:24:40.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:24:40.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:24:40.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:24:40.800 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:40.798+0000 7f0558c687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:24:40.832 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:40.830+0000 7f0558c687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:24:40.833 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:40.830+0000 7f0558c687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:24:41.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:24:41.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:24:41.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:24:41.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:24:41.205 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:24:41.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:24:41.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:41.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:24:41.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:41.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:41.412 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:41.410+0000 7f0558c687c0 -1 Falling back to public interface 2024-09-16T17:24:41.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:42.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:42.642 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:24:42.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:42.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:24:42.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:42.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:43.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:43.991 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:24:43.990+0000 7f0558c687c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:24:44.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:44.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:44.079 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:24:44.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:24:44.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:44.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:44.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:45.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:45.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:45.635 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:24:45.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:24:45.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:45.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:46.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:47.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:47.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:47.100 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:24:47.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:24:47.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:47.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:47.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:48.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:48.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:48.572 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:24:48.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:24:48.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:48.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:49.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:50.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:50.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:50.133 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:24:50.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:24:50.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:50.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:50.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:51.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:51.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:51.582 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:24:51.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:24:51.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:51.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:52.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:53.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:53.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:53.124 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:24:53.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:24:53.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:53.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:53.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:54.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:54.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:54.562 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:24:54.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:24:54.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:54.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:56.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:56.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:56.098 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:24:56.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:24:56.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:56.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:56.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:57.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:57.537 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:24:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:24:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:57.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:24:58.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:24:59.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:24:59.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:24:59.310 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:24:59.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:24:59.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:24:59.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:25:00.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:01.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:01.046 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:25:01.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:01.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:25:01.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:01.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:25:01.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:02.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:02.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:02.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:25:02.874 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:25:02.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:02.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:25:04.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:05.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:05.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:05.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:25:05.093 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:25:05.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:05.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:25:05.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:06.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:06.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:06.923 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:25:06.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:25:06.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:06.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:25:07.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:08.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:08.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:08.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:25:08.851 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:25:08.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:08.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:25:09.296 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3853346333,v1:127.0.0.1:6803/3853346333] [v2:127.0.0.1:6804/3853346333,v1:127.0.0.1:6805/3853346333] exists,up 074c2305-9519-4c47-89d0-96827be4a349 2024-09-16T17:25:09.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:25:09.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:25:09.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:25:09.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-16T17:25:09.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 1 2024-09-16T17:25:09.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-16T17:25:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:25:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T17:25:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:25:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/1 2024-09-16T17:25:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T17:25:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:25:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:25:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:25:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/1' 2024-09-16T17:25:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/1/journal' 2024-09-16T17:25:09.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:25:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:25:09.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-16T17:25:09.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:25:09.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:25:09.300 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:25:09.301 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:25:09.301 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:25:09.302 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:25:09.302 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:25:09.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:25:09.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:25:09.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:25:09.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:25:09.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-16T17:25:09.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-16T17:25:09.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:25:09.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:25:09.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:25:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:25:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:25:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:25:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/1 2024-09-16T17:25:09.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:25:09.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0f523a50-8960-40c0-b6f8-73b771a39242 2024-09-16T17:25:09.303 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 0f523a50-8960-40c0-b6f8-73b771a39242 2024-09-16T17:25:09.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0f523a50-8960-40c0-b6f8-73b771a39242' 2024-09-16T17:25:09.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:25:09.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1aehmSDu8ExAAgonE+IjhaTYFn+4r6OUovw== 2024-09-16T17:25:09.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1aehmSDu8ExAAgonE+IjhaTYFn+4r6OUovw=="}' 2024-09-16T17:25:09.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0f523a50-8960-40c0-b6f8-73b771a39242 -i td/repeer-on-acting-back/1/new.json 2024-09-16T17:25:10.075 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:25:10.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/1/new.json 2024-09-16T17:25:10.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1aehmSDu8ExAAgonE+IjhaTYFn+4r6OUovw== --osd-uuid 0f523a50-8960-40c0-b6f8-73b771a39242 2024-09-16T17:25:10.131 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:10.129+0000 7f1c0a81d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:10.155 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:10.153+0000 7f1c0a81d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:10.157 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:10.153+0000 7f1c0a81d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:10.157 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:10.153+0000 7f1c0a81d7c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-09-16T17:25:10.157 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:10.153+0000 7f1c0a81d7c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-09-16T17:25:10.157 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:10.153+0000 7f1c0a81d7c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-09-16T17:25:10.255 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:10.253+0000 7f1c0a81d7c0 -1 bluestore(td/repeer-on-acting-back/1) _read_fsid unparsable uuid 2024-09-16T17:25:16.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/1/keyring 2024-09-16T17:25:16.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:25:16.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T17:25:16.398 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T17:25:16.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:25:16.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T17:25:16.972 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:25:16.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:25:16.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:25:16.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:25:16.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:25:17.012 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:17.009+0000 7fc53e9b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:17.045 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:17.041+0000 7fc53e9b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:17.046 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:17.041+0000 7fc53e9b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:17.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T17:25:17.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:25:17.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:25:17.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:25:17.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:25:17.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:17.431 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:25:17.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:25:17.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:17.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:17.626 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:17.621+0000 7fc53e9b17c0 -1 Falling back to public interface 2024-09-16T17:25:17.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:18.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:18.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:18.909 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:25:18.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:25:18.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:18.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:19.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:20.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:20.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:20.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:25:20.346 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:25:20.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:20.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:20.395 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:20.393+0000 7fc53e9b17c0 -1 osd.1 0 log_to_monitors true 2024-09-16T17:25:20.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:21.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:21.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:21.900 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:25:21.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:25:21.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:21.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:22.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:23.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:23.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:23.520 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:25:23.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:25:23.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:23.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:23.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:24.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:24.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:24.947 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:25:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:25:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:25.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:26.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:26.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:26.688 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:25:26.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:25:26.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:26.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:27.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:28.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:28.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:28.156 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:25:28.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:25:28.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:28.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:28.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:29.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:29.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:29.899 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:25:29.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:25:29.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:29.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:30.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:31.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:31.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:31.533 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:25:31.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:25:31.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:31.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:32.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:33.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:33.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:33.026 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:25:33.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:25:33.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:33.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:33.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:34.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:34.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:34.533 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:25:34.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:25:34.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:34.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:35.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:36.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:36.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:36.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:25:36.104 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:25:36.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:36.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:36.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:38.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:38.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:38.002 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:25:38.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:25:38.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:38.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:39.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:40.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:40.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:40.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:25:40.427 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:25:40.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:40.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:41.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:42.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:42.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:42.464 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:25:42.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:25:42.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:42.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:43.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:44.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:44.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:44.354 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:25:44.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:25:44.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:44.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:45.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:46.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:46.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:46.530 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:25:46.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:25:46.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:46.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:47.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:48.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:48.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:48.098 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:25:48.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T17:25:48.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:48.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:25:48.546 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3876240386,v1:127.0.0.1:6811/3876240386] [v2:127.0.0.1:6812/3876240386,v1:127.0.0.1:6813/3876240386] exists,up 0f523a50-8960-40c0-b6f8-73b771a39242 2024-09-16T17:25:48.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:25:48.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:25:48.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:25:48.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-16T17:25:48.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 2 2024-09-16T17:25:48.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/2 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:25:48.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:25:48.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-16T17:25:48.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-16T17:25:48.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:25:48.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:25:48.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:25:48.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:25:48.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:25:48.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:25:48.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/2 2024-09-16T17:25:48.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:25:48.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3c955f00-9762-4991-85b7-355da0a2effa 2024-09-16T17:25:48.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3c955f00-9762-4991-85b7-355da0a2effa' 2024-09-16T17:25:48.553 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 3c955f00-9762-4991-85b7-355da0a2effa 2024-09-16T17:25:48.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:25:48.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcauhm7ZEXIhAAF2AbcSOTvIBgLT312dVgEA== 2024-09-16T17:25:48.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcauhm7ZEXIhAAF2AbcSOTvIBgLT312dVgEA=="}' 2024-09-16T17:25:48.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3c955f00-9762-4991-85b7-355da0a2effa -i td/repeer-on-acting-back/2/new.json 2024-09-16T17:25:49.093 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:25:49.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/2/new.json 2024-09-16T17:25:49.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAcauhm7ZEXIhAAF2AbcSOTvIBgLT312dVgEA== --osd-uuid 3c955f00-9762-4991-85b7-355da0a2effa 2024-09-16T17:25:49.144 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:49.140+0000 7fe571d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:49.147 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:49.144+0000 7fe571d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:49.148 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:49.144+0000 7fe571d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:49.148 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:49.144+0000 7fe571d1f7c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-09-16T17:25:49.148 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:49.144+0000 7fe571d1f7c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-09-16T17:25:49.148 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:49.144+0000 7fe571d1f7c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-09-16T17:25:49.186 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:49.184+0000 7fe571d1f7c0 -1 bluestore(td/repeer-on-acting-back/2) _read_fsid unparsable uuid 2024-09-16T17:25:55.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/2/keyring 2024-09-16T17:25:55.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:25:55.634 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T17:25:55.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T17:25:55.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:25:56.218 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T17:25:56.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T17:25:56.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:25:56.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:25:56.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:25:56.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:25:56.258 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:56.252+0000 7f9342a4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:56.339 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:56.336+0000 7f9342a4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:56.340 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:56.336+0000 7f9342a4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:25:56.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T17:25:56.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:25:56.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:25:56.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:25:56.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:25:56.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:56.683 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:25:56.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:25:56.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:56.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:25:56.917 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:56.912+0000 7f9342a4e7c0 -1 Falling back to public interface 2024-09-16T17:25:57.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:58.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:58.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:58.130 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:25:58.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:25:58.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:58.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:25:58.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:25:59.554 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:25:59.548+0000 7f9342a4e7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T17:25:59.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:25:59.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:25:59.578 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:25:59.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:25:59.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:25:59.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:00.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:01.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:01.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:01.133 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:26:01.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:26:01.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:01.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:01.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:02.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:02.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:02.592 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:26:02.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:26:02.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:02.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:04.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:04.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:04.067 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:26:04.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:26:04.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:04.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:04.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:05.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:05.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:05.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:26:05.606 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:26:05.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:05.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:06.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:07.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:07.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:07.064 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:26:07.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:26:07.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:07.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:07.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:08.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:08.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:08.665 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:26:08.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:26:08.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:08.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:10.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:10.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:10.232 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:26:10.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:26:10.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:10.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:10.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:11.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:11.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:11.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:26:11.794 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:26:11.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:11.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:12.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:13.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:13.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:13.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:26:13.250 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:26:13.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:13.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:13.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:14.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:14.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:14.713 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:26:14.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:26:14.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:14.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:15.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:16.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:16.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:16.717 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:26:16.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:26:16.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:16.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:17.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:18.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:18.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:18.504 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:26:18.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:26:18.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:18.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:19.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:20.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:20.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:20.364 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:26:20.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:26:20.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:20.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:20.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:21.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:21.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:21.812 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:26:21.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:26:21.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:21.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:22.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:23.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:23.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:23.578 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:26:23.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:26:23.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:23.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:24.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:25.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:25.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:25.316 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:26:25.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T17:26:25.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:25.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:26:25.846 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3146885453,v1:127.0.0.1:6819/3146885453] [v2:127.0.0.1:6820/3146885453,v1:127.0.0.1:6821/3146885453] exists,up 3c955f00-9762-4991-85b7-355da0a2effa 2024-09-16T17:26:25.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:26:25.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:26:25.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:26:25.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-16T17:26:25.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 3 2024-09-16T17:26:25.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-16T17:26:25.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/3 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/3' 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/3/journal' 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:26:25.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:26:25.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:26:25.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:26:25.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/3 2024-09-16T17:26:25.852 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:26:25.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=095adcd8-9d8f-4586-92c0-8b4c738b0fc8 2024-09-16T17:26:25.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 095adcd8-9d8f-4586-92c0-8b4c738b0fc8' 2024-09-16T17:26:25.853 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 095adcd8-9d8f-4586-92c0-8b4c738b0fc8 2024-09-16T17:26:25.853 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:26:25.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBauhmCmsQNBAAep2Oth5vfgJyP+5TDa/Keg== 2024-09-16T17:26:25.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBauhmCmsQNBAAep2Oth5vfgJyP+5TDa/Keg=="}' 2024-09-16T17:26:25.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 095adcd8-9d8f-4586-92c0-8b4c738b0fc8 -i td/repeer-on-acting-back/3/new.json 2024-09-16T17:26:26.419 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:26:26.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/3/new.json 2024-09-16T17:26:26.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBauhmCmsQNBAAep2Oth5vfgJyP+5TDa/Keg== --osd-uuid 095adcd8-9d8f-4586-92c0-8b4c738b0fc8 2024-09-16T17:26:26.478 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:26.471+0000 7f528ce557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:26:26.508 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:26.503+0000 7f528ce557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:26:26.510 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:26.503+0000 7f528ce557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:26:26.510 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:26.507+0000 7f528ce557c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-09-16T17:26:26.511 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:26.507+0000 7f528ce557c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-09-16T17:26:26.511 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:26.507+0000 7f528ce557c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-09-16T17:26:26.600 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:26.595+0000 7f528ce557c0 -1 bluestore(td/repeer-on-acting-back/3) _read_fsid unparsable uuid 2024-09-16T17:26:32.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/3/keyring 2024-09-16T17:26:32.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:26:32.613 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-16T17:26:32.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T17:26:32.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:26:33.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T17:26:33.197 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-16T17:26:33.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:26:33.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:26:33.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:26:33.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:26:33.238 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:33.231+0000 7f0d688b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:26:33.251 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:33.247+0000 7f0d688b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:26:33.253 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:33.247+0000 7f0d688b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:26:33.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T17:26:33.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:26:33.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T17:26:33.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:26:33.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:26:33.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:33.637 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:26:33.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:26:33.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:33.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:33.824 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:33.819+0000 7f0d688b17c0 -1 Falling back to public interface 2024-09-16T17:26:34.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:35.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:35.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:35.094 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:26:35.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:26:35.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:35.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:35.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:36.355 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:26:36.351+0000 7f0d688b17c0 -1 osd.3 0 log_to_monitors true 2024-09-16T17:26:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:36.571 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:26:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:26:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:37.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:38.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:38.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:38.045 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:26:38.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:26:38.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:38.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:38.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:39.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:39.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:39.480 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:26:39.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:26:39.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:39.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:39.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:40.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:40.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:40.948 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:26:40.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:26:40.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:40.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:41.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:42.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:42.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:42.470 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:26:42.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:26:42.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:42.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:43.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:44.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:44.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:44.176 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:26:44.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:26:44.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:44.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:45.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:45.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:45.695 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:26:45.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:26:45.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:45.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:46.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:47.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:47.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:47.186 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:26:47.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:26:47.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:47.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:47.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:48.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:48.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:48.648 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:26:48.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:26:48.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:48.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:49.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:50.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:26:50.498 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:26:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:50.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:51.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:52.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:52.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:52.179 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:26:52.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:26:52.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:52.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:52.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:53.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:53.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:53.675 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:26:53.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:26:53.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:53.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:54.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:55.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:55.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:55.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:26:55.405 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:26:55.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:55.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:56.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:57.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:57.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:57.057 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:26:57.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:26:57.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:57.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:57.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:26:58.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:26:58.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:26:58.642 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:26:58.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:26:58.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:26:58.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:26:59.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:00.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:00.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:00.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:27:00.348 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:27:00.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:00.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T17:27:00.817 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4034569082,v1:127.0.0.1:6827/4034569082] [v2:127.0.0.1:6828/4034569082,v1:127.0.0.1:6829/4034569082] exists,up 095adcd8-9d8f-4586-92c0-8b4c738b0fc8 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 4 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/4 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T17:27:00.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/4' 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/4/journal' 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:27:00.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:27:00.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/4 2024-09-16T17:27:00.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:27:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8336bb89-4fb4-44df-bee3-abfb687d8e92 2024-09-16T17:27:00.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 8336bb89-4fb4-44df-bee3-abfb687d8e92' 2024-09-16T17:27:00.824 INFO:tasks.workunit.client.0.smithi039.stdout:add osd4 8336bb89-4fb4-44df-bee3-abfb687d8e92 2024-09-16T17:27:00.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:27:00.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBkauhmqU+rMhAA7Gxw0ddznvU35nTu0GuCVQ== 2024-09-16T17:27:00.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBkauhmqU+rMhAA7Gxw0ddznvU35nTu0GuCVQ=="}' 2024-09-16T17:27:00.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8336bb89-4fb4-44df-bee3-abfb687d8e92 -i td/repeer-on-acting-back/4/new.json 2024-09-16T17:27:01.412 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:27:01.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/4/new.json 2024-09-16T17:27:01.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkauhmqU+rMhAA7Gxw0ddznvU35nTu0GuCVQ== --osd-uuid 8336bb89-4fb4-44df-bee3-abfb687d8e92 2024-09-16T17:27:01.470 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:01.463+0000 7f46f81897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:01.493 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:01.491+0000 7f46f81897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:01.495 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:01.491+0000 7f46f81897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:01.495 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:01.491+0000 7f46f81897c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-09-16T17:27:01.495 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:01.491+0000 7f46f81897c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-09-16T17:27:01.496 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:01.491+0000 7f46f81897c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-09-16T17:27:01.585 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:01.583+0000 7f46f81897c0 -1 bluestore(td/repeer-on-acting-back/4) _read_fsid unparsable uuid 2024-09-16T17:27:07.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/4/keyring 2024-09-16T17:27:07.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:27:07.686 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd4 key to auth repository 2024-09-16T17:27:07.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T17:27:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:27:08.259 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.4 2024-09-16T17:27:08.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T17:27:08.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:27:08.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:27:08.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:27:08.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:27:08.298 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:08.295+0000 7f275e10a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:08.309 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:08.307+0000 7f275e10a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:08.311 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:08.311+0000 7f275e10a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:08.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T17:27:08.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:27:08.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T17:27:08.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:27:08.698 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:27:08.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:27:08.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:08.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:27:08.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:08.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:08.867+0000 7f275e10a7c0 -1 Falling back to public interface 2024-09-16T17:27:09.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:10.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:10.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:10.150 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:27:10.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:27:10.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:10.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:10.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:11.421 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:11.419+0000 7f275e10a7c0 -1 osd.4 0 log_to_monitors true 2024-09-16T17:27:11.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:11.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:11.597 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:27:11.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:27:11.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:11.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:12.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:13.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:13.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:13.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:27:13.164 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:27:13.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:13.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:13.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:14.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:14.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:14.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:27:14.614 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:27:14.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:14.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:15.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:16.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:16.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:16.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:27:16.056 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:27:16.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:16.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:16.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:17.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:17.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:17.762 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:27:17.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:27:17.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:17.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:18.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:19.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:19.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:19.381 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:27:19.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:27:19.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:19.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:19.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:20.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:20.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:20.859 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:27:20.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:27:20.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:20.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:21.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:22.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:22.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:22.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:27:22.357 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:27:22.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:22.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:22.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:23.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:23.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:23.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:27:23.969 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:27:23.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:23.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:24.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:25.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:25.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:25.946 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:27:25.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:27:25.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:25.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:26.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:27.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:27.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:27.984 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:27:27.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:27:27.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:27.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:28.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:29.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:29.679 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:27:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:27:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:29.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:30.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:31.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:31.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:31.551 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:27:31.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:27:31.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:31.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:32.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:33.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:33.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:33.274 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:27:33.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:27:33.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:33.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:33.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:34.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:34.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:34.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:27:34.778 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:27:34.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:34.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:35.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:36.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:36.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:36.286 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:27:36.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:27:36.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:36.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T17:27:36.746 INFO:tasks.workunit.client.0.smithi039.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/779917338,v1:127.0.0.1:6835/779917338] [v2:127.0.0.1:6836/779917338,v1:127.0.0.1:6837/779917338] exists,up 8336bb89-4fb4-44df-bee3-abfb687d8e92 2024-09-16T17:27:36.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:27:36.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:27:36.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:27:36.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-16T17:27:36.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 5 2024-09-16T17:27:36.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-16T17:27:36.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:27:36.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T17:27:36.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:27:36.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/5 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/5' 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/5/journal' 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:27:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:27:36.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/5 2024-09-16T17:27:36.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:27:36.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5cd89463-0d1f-4ab5-b74c-285ba5e2549b 2024-09-16T17:27:36.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 5cd89463-0d1f-4ab5-b74c-285ba5e2549b' 2024-09-16T17:27:36.753 INFO:tasks.workunit.client.0.smithi039.stdout:add osd5 5cd89463-0d1f-4ab5-b74c-285ba5e2549b 2024-09-16T17:27:36.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:27:36.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIauhm7md1LhAAOx8h8L0OU7b1Z+GJ+zpYSw== 2024-09-16T17:27:36.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIauhm7md1LhAAOx8h8L0OU7b1Z+GJ+zpYSw=="}' 2024-09-16T17:27:36.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5cd89463-0d1f-4ab5-b74c-285ba5e2549b -i td/repeer-on-acting-back/5/new.json 2024-09-16T17:27:37.326 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:27:37.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/5/new.json 2024-09-16T17:27:37.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIauhm7md1LhAAOx8h8L0OU7b1Z+GJ+zpYSw== --osd-uuid 5cd89463-0d1f-4ab5-b74c-285ba5e2549b 2024-09-16T17:27:37.385 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:37.382+0000 7f2b669c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:37.388 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:37.386+0000 7f2b669c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:37.389 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:37.386+0000 7f2b669c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:37.389 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:37.386+0000 7f2b669c37c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-09-16T17:27:37.389 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:37.386+0000 7f2b669c37c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-09-16T17:27:37.390 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:37.386+0000 7f2b669c37c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-09-16T17:27:37.500 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:37.498+0000 7f2b669c37c0 -1 bluestore(td/repeer-on-acting-back/5) _read_fsid unparsable uuid 2024-09-16T17:27:43.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/5/keyring 2024-09-16T17:27:43.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:27:43.528 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd5 key to auth repository 2024-09-16T17:27:43.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T17:27:43.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:27:44.107 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.5 2024-09-16T17:27:44.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T17:27:44.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:27:44.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:27:44.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:27:44.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:27:44.147 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:44.146+0000 7f174e5f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:44.156 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:44.154+0000 7f174e5f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:44.157 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:44.154+0000 7f174e5f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:27:44.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T17:27:44.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:27:44.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T17:27:44.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:27:44.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:27:44.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:44.599 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:27:44.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:27:44.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:44.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:27:44.736 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:44.734+0000 7f174e5f17c0 -1 Falling back to public interface 2024-09-16T17:27:45.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:46.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:46.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:46.024 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:27:46.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:27:46.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:46.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:27:46.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:47.269 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:27:47.266+0000 7f174e5f17c0 -1 osd.5 0 log_to_monitors true 2024-09-16T17:27:47.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:47.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:47.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:27:47.504 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:27:47.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:47.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:27:47.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:48.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:48.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:48.976 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:27:48.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:27:48.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:48.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:27:49.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:50.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:50.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:50.467 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:27:50.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:27:50.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:50.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:27:50.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:51.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:51.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:51.962 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:27:51.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:27:51.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:51.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:27:52.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:53.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:53.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:53.394 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:27:53.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:27:53.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:53.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:27:53.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:54.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:54.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:54.860 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:27:54.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:27:54.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:54.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:27:55.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:56.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:56.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:56.422 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:27:56.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:27:56.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:56.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:27:56.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:57.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:57.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:57.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:27:57.868 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:27:57.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:57.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:27:58.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:27:59.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:27:59.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:27:59.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:27:59.426 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:27:59.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:27:59.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:27:59.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:28:00.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:28:00.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:28:00.923 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:28:00.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:28:00.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:28:00.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:28:01.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:28:02.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:28:02.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:28:02.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:28:02.704 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:28:02.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:28:02.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:28:03.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:28:04.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:28:04.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:28:04.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:28:04.602 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:28:04.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:28:04.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:28:05.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:28:06.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:28:06.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:28:06.365 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:28:06.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:28:06.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:28:06.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:28:07.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:28:08.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:28:08.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:28:08.064 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:28:08.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:28:08.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:28:08.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:28:09.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:28:10.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:28:10.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:28:10.177 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:28:10.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:28:10.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:28:10.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:28:10.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:28:11.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:28:11.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:28:11.712 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:28:11.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:28:11.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:28:11.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T17:28:12.150 INFO:tasks.workunit.client.0.smithi039.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3844807403,v1:127.0.0.1:6843/3844807403] [v2:127.0.0.1:6844/3844807403,v1:127.0.0.1:6845/3844807403] exists,up 5cd89463-0d1f-4ab5-b74c-285ba5e2549b 2024-09-16T17:28:12.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:28:12.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:28:12.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:28:12.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:60: TEST_repeer_on_down_acting_member_coming_back: create_pool test 1 1 2024-09-16T17:28:12.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-16T17:28:12.898 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T17:28:12.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T17:28:13.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:61: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test size 3 2024-09-16T17:28:14.779 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 3 2024-09-16T17:28:14.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:62: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test min_size 2 2024-09-16T17:28:15.641 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 2 2024-09-16T17:28:15.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: ceph pg dump pools -f json 2024-09-16T17:28:15.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq .pool_stats 2024-09-16T17:28:15.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq '.[].poolid' 2024-09-16T17:28:16.084 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pools 2024-09-16T17:28:16.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: local poolid=1 2024-09-16T17:28:16.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:64: TEST_repeer_on_down_acting_member_coming_back: local pgid=1.0 2024-09-16T17:28:16.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:67: TEST_repeer_on_down_acting_member_coming_back: ceph osd set-require-min-compat-client luminous 2024-09-16T17:28:16.997 INFO:tasks.workunit.client.0.smithi039.stderr:set require_min_compat_client to luminous 2024-09-16T17:28:17.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:69: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 2 3 2024-09-16T17:28:18.600 INFO:tasks.workunit.client.0.smithi039.stderr:set 1.0 pg_upmap mapping to [1,2,3] 2024-09-16T17:28:18.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:71: TEST_repeer_on_down_acting_member_coming_back: flush_pg_stats 2024-09-16T17:28:18.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:28:18.623 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:28:19.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:28:19.165 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:28:19.165 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:28:19.165 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T17:28:19.165 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T17:28:19.165 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T17:28:19.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:28:19.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:19.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:28:19.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836520 2024-09-16T17:28:19.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836520 2024-09-16T17:28:19.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520' 2024-09-16T17:28:19.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:19.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:28:19.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672999 2024-09-16T17:28:19.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672999 2024-09-16T17:28:19.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520 1-42949672999' 2024-09-16T17:28:19.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:19.502 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:28:19.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509470 2024-09-16T17:28:19.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509470 2024-09-16T17:28:19.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509470' 2024-09-16T17:28:19.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:19.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:28:19.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345939 2024-09-16T17:28:19.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345939 2024-09-16T17:28:19.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509470 3-85899345939' 2024-09-16T17:28:19.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:19.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T17:28:19.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182416 2024-09-16T17:28:19.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182416 2024-09-16T17:28:19.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509470 3-85899345939 4-107374182416' 2024-09-16T17:28:19.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:19.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T17:28:20.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018891 2024-09-16T17:28:20.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018891 2024-09-16T17:28:20.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509470 3-85899345939 4-107374182416 5-128849018891' 2024-09-16T17:28:20.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:20.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836520 2024-09-16T17:28:20.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:20.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:28:20.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836520 2024-09-16T17:28:20.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:20.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836520 2024-09-16T17:28:20.153 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836520 2024-09-16T17:28:20.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-09-16T17:28:20.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:28:20.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836521 -lt 21474836520 2024-09-16T17:28:20.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:20.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672999 2024-09-16T17:28:20.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:20.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:28:20.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672999 2024-09-16T17:28:20.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:20.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672999 2024-09-16T17:28:20.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2024-09-16T17:28:20.610 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672999 2024-09-16T17:28:20.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:28:21.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672999 -lt 42949672999 2024-09-16T17:28:21.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:21.031 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509470 2024-09-16T17:28:21.032 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:21.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:28:21.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509470 2024-09-16T17:28:21.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:21.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509470 2024-09-16T17:28:21.036 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509470 2024-09-16T17:28:21.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-09-16T17:28:21.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:28:21.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509470 -lt 64424509470 2024-09-16T17:28:21.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:21.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345939 2024-09-16T17:28:21.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:21.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:28:21.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:21.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345939 2024-09-16T17:28:21.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345939 2024-09-16T17:28:21.536 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345939 2024-09-16T17:28:21.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2024-09-16T17:28:21.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:28:21.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345940 -lt 85899345939 2024-09-16T17:28:21.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:21.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182416 2024-09-16T17:28:21.969 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:21.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T17:28:21.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182416 2024-09-16T17:28:21.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:21.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182416 2024-09-16T17:28:21.973 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182416 2024-09-16T17:28:21.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182416' 2024-09-16T17:28:21.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T17:28:22.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182417 -lt 107374182416 2024-09-16T17:28:22.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:22.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018891 2024-09-16T17:28:22.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:22.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T17:28:22.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018891 2024-09-16T17:28:22.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:22.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018891 2024-09-16T17:28:22.488 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018891 2024-09-16T17:28:22.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018891' 2024-09-16T17:28:22.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T17:28:22.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018892 -lt 128849018891 2024-09-16T17:28:22.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:72: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-09-16T17:28:22.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:28:22.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:28:22.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:28:22.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:28:22.922 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:28:22.923 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:28:22.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:28:22.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:28:22.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:28:23.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:28:23.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:28:23.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:28:23.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:28:23.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:28:23.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:28:23.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:28:23.448 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:28:23.448 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:28:23.448 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T17:28:23.448 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T17:28:23.448 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T17:28:23.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:28:23.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:23.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:28:23.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836523 2024-09-16T17:28:23.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836523 2024-09-16T17:28:23.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523' 2024-09-16T17:28:23.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:23.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:28:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673002 2024-09-16T17:28:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673002 2024-09-16T17:28:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949673002' 2024-09-16T17:28:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:23.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:28:23.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509473 2024-09-16T17:28:23.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509473 2024-09-16T17:28:23.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-64424509473' 2024-09-16T17:28:23.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:23.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:28:24.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345942 2024-09-16T17:28:24.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345942 2024-09-16T17:28:24.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-64424509473 3-85899345942' 2024-09-16T17:28:24.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:24.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T17:28:24.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182419 2024-09-16T17:28:24.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182419 2024-09-16T17:28:24.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-64424509473 3-85899345942 4-107374182419' 2024-09-16T17:28:24.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:24.261 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T17:28:24.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018894 2024-09-16T17:28:24.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018894 2024-09-16T17:28:24.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-64424509473 3-85899345942 4-107374182419 5-128849018894' 2024-09-16T17:28:24.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:24.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836523 2024-09-16T17:28:24.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:24.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:28:24.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836523 2024-09-16T17:28:24.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:24.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836523 2024-09-16T17:28:24.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-09-16T17:28:24.429 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836523 2024-09-16T17:28:24.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:28:24.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836523 -lt 21474836523 2024-09-16T17:28:24.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:24.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673002 2024-09-16T17:28:24.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:24.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:28:24.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673002 2024-09-16T17:28:24.901 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:24.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673002 2024-09-16T17:28:24.903 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673002 2024-09-16T17:28:24.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2024-09-16T17:28:24.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:28:25.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673002 -lt 42949673002 2024-09-16T17:28:25.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:25.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:25.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509473 2024-09-16T17:28:25.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:28:25.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:25.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509473 2024-09-16T17:28:25.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509473 2024-09-16T17:28:25.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-09-16T17:28:25.339 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509473 2024-09-16T17:28:25.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:28:25.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509473 -lt 64424509473 2024-09-16T17:28:25.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:25.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345942 2024-09-16T17:28:25.751 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:25.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:28:25.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345942 2024-09-16T17:28:25.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:25.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345942 2024-09-16T17:28:25.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2024-09-16T17:28:25.755 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345942 2024-09-16T17:28:25.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:28:26.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345942 -lt 85899345942 2024-09-16T17:28:26.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:26.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182419 2024-09-16T17:28:26.170 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:26.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T17:28:26.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182419 2024-09-16T17:28:26.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:26.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182419 2024-09-16T17:28:26.173 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182419 2024-09-16T17:28:26.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182419' 2024-09-16T17:28:26.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T17:28:26.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182420 -lt 107374182419 2024-09-16T17:28:26.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:26.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018894 2024-09-16T17:28:26.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:26.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T17:28:26.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018894 2024-09-16T17:28:26.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:26.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018894 2024-09-16T17:28:26.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018894' 2024-09-16T17:28:26.612 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018894 2024-09-16T17:28:26.612 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T17:28:27.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018894 -lt 128849018894 2024-09-16T17:28:27.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:28:27.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:28:27.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:28:27.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T17:28:27.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:28:27.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:28:27.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:28:27.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:28:27.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:28:27.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:28:27.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:28:28.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T17:28:28.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:28:28.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:28:28.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:28:28.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T17:28:28.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:28:28.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:28:28.621 INFO:tasks.workunit.client.0.smithi039.stdout:writing initial objects 2024-09-16T17:28:28.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:74: TEST_repeer_on_down_acting_member_coming_back: echo 'writing initial objects' 2024-09-16T17:28:28.621 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: seq 1 100 2024-09-16T17:28:28.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:28.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_1 /etc/fstab 2024-09-16T17:28:28.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:28.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_2 /etc/fstab 2024-09-16T17:28:28.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:28.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_3 /etc/fstab 2024-09-16T17:28:29.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:29.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_4 /etc/fstab 2024-09-16T17:28:29.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:29.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_5 /etc/fstab 2024-09-16T17:28:29.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:29.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_6 /etc/fstab 2024-09-16T17:28:29.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:29.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_7 /etc/fstab 2024-09-16T17:28:29.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:29.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_8 /etc/fstab 2024-09-16T17:28:30.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:30.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_9 /etc/fstab 2024-09-16T17:28:30.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:30.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_10 /etc/fstab 2024-09-16T17:28:30.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:30.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_11 /etc/fstab 2024-09-16T17:28:30.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:30.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_12 /etc/fstab 2024-09-16T17:28:30.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:30.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_13 /etc/fstab 2024-09-16T17:28:31.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:31.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_14 /etc/fstab 2024-09-16T17:28:31.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:31.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_15 /etc/fstab 2024-09-16T17:28:31.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:31.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_16 /etc/fstab 2024-09-16T17:28:31.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:31.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_17 /etc/fstab 2024-09-16T17:28:31.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:31.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_18 /etc/fstab 2024-09-16T17:28:31.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:31.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_19 /etc/fstab 2024-09-16T17:28:32.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:32.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_20 /etc/fstab 2024-09-16T17:28:32.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:32.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_21 /etc/fstab 2024-09-16T17:28:32.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:32.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_22 /etc/fstab 2024-09-16T17:28:32.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:32.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_23 /etc/fstab 2024-09-16T17:28:32.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:32.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_24 /etc/fstab 2024-09-16T17:28:33.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:33.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_25 /etc/fstab 2024-09-16T17:28:33.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:33.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_26 /etc/fstab 2024-09-16T17:28:33.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:33.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_27 /etc/fstab 2024-09-16T17:28:33.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:33.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_28 /etc/fstab 2024-09-16T17:28:33.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:33.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_29 /etc/fstab 2024-09-16T17:28:33.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:33.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_30 /etc/fstab 2024-09-16T17:28:34.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:34.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_31 /etc/fstab 2024-09-16T17:28:34.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:34.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_32 /etc/fstab 2024-09-16T17:28:34.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:34.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_33 /etc/fstab 2024-09-16T17:28:34.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:34.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_34 /etc/fstab 2024-09-16T17:28:34.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:34.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_35 /etc/fstab 2024-09-16T17:28:34.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:34.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_36 /etc/fstab 2024-09-16T17:28:35.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:35.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_37 /etc/fstab 2024-09-16T17:28:35.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:35.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_38 /etc/fstab 2024-09-16T17:28:35.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:35.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_39 /etc/fstab 2024-09-16T17:28:35.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:35.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_40 /etc/fstab 2024-09-16T17:28:35.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:35.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_41 /etc/fstab 2024-09-16T17:28:36.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:36.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_42 /etc/fstab 2024-09-16T17:28:36.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:36.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_43 /etc/fstab 2024-09-16T17:28:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_44 /etc/fstab 2024-09-16T17:28:36.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:36.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_45 /etc/fstab 2024-09-16T17:28:36.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:36.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_46 /etc/fstab 2024-09-16T17:28:36.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:36.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_47 /etc/fstab 2024-09-16T17:28:37.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:37.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_48 /etc/fstab 2024-09-16T17:28:37.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:37.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_49 /etc/fstab 2024-09-16T17:28:37.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:37.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_50 /etc/fstab 2024-09-16T17:28:37.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:37.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_51 /etc/fstab 2024-09-16T17:28:37.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:37.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_52 /etc/fstab 2024-09-16T17:28:37.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:37.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_53 /etc/fstab 2024-09-16T17:28:38.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:38.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_54 /etc/fstab 2024-09-16T17:28:38.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:38.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_55 /etc/fstab 2024-09-16T17:28:38.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:38.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_56 /etc/fstab 2024-09-16T17:28:38.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:38.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_57 /etc/fstab 2024-09-16T17:28:38.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:38.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_58 /etc/fstab 2024-09-16T17:28:39.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:39.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_59 /etc/fstab 2024-09-16T17:28:39.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:39.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_60 /etc/fstab 2024-09-16T17:28:39.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:39.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_61 /etc/fstab 2024-09-16T17:28:39.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:39.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_62 /etc/fstab 2024-09-16T17:28:39.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:39.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_63 /etc/fstab 2024-09-16T17:28:39.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:39.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_64 /etc/fstab 2024-09-16T17:28:40.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:40.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_65 /etc/fstab 2024-09-16T17:28:40.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:40.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_66 /etc/fstab 2024-09-16T17:28:40.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:40.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_67 /etc/fstab 2024-09-16T17:28:40.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:40.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_68 /etc/fstab 2024-09-16T17:28:40.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:40.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_69 /etc/fstab 2024-09-16T17:28:40.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:40.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_70 /etc/fstab 2024-09-16T17:28:41.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:41.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_71 /etc/fstab 2024-09-16T17:28:41.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:41.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_72 /etc/fstab 2024-09-16T17:28:41.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:41.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_73 /etc/fstab 2024-09-16T17:28:41.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:41.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_74 /etc/fstab 2024-09-16T17:28:41.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:41.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_75 /etc/fstab 2024-09-16T17:28:42.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:42.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_76 /etc/fstab 2024-09-16T17:28:42.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:42.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_77 /etc/fstab 2024-09-16T17:28:42.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:42.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_78 /etc/fstab 2024-09-16T17:28:42.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:42.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_79 /etc/fstab 2024-09-16T17:28:42.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:42.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_80 /etc/fstab 2024-09-16T17:28:42.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:42.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_81 /etc/fstab 2024-09-16T17:28:43.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:43.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_82 /etc/fstab 2024-09-16T17:28:43.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:43.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_83 /etc/fstab 2024-09-16T17:28:43.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:43.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_84 /etc/fstab 2024-09-16T17:28:43.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:43.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_85 /etc/fstab 2024-09-16T17:28:43.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:43.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_86 /etc/fstab 2024-09-16T17:28:43.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:43.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_87 /etc/fstab 2024-09-16T17:28:44.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:44.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_88 /etc/fstab 2024-09-16T17:28:44.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:44.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_89 /etc/fstab 2024-09-16T17:28:44.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:44.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_90 /etc/fstab 2024-09-16T17:28:44.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:44.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_91 /etc/fstab 2024-09-16T17:28:44.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:44.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_92 /etc/fstab 2024-09-16T17:28:44.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:44.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_93 /etc/fstab 2024-09-16T17:28:45.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:45.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_94 /etc/fstab 2024-09-16T17:28:45.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:45.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_95 /etc/fstab 2024-09-16T17:28:45.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:45.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_96 /etc/fstab 2024-09-16T17:28:45.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:45.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_97 /etc/fstab 2024-09-16T17:28:45.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:45.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_98 /etc/fstab 2024-09-16T17:28:46.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:46.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_99 /etc/fstab 2024-09-16T17:28:46.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-16T17:28:46.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_100 /etc/fstab 2024-09-16T17:28:46.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T17:28:46.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-09-16T17:28:46.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:28:46.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:28:46.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:28:46.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T17:28:46.421 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:28:46.421 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:28:46.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:28:46.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:28:46.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:28:46.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T17:28:46.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:28:46.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:28:46.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:28:46.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:28:46.499 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:28:46.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:28:46.937 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:28:46.937 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:28:46.937 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T17:28:46.937 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T17:28:46.937 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T17:28:46.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:28:46.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:46.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:28:47.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836530 2024-09-16T17:28:47.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836530 2024-09-16T17:28:47.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836530' 2024-09-16T17:28:47.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:47.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:28:47.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673008 2024-09-16T17:28:47.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673008 2024-09-16T17:28:47.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836530 1-42949673008' 2024-09-16T17:28:47.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:47.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:28:47.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509480 2024-09-16T17:28:47.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509480 2024-09-16T17:28:47.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509480' 2024-09-16T17:28:47.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:47.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:28:47.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345949 2024-09-16T17:28:47.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345949 2024-09-16T17:28:47.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509480 3-85899345949' 2024-09-16T17:28:47.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:47.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T17:28:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182426 2024-09-16T17:28:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182426 2024-09-16T17:28:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509480 3-85899345949 4-107374182426' 2024-09-16T17:28:47.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:28:47.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T17:28:47.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018901 2024-09-16T17:28:47.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018901 2024-09-16T17:28:47.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509480 3-85899345949 4-107374182426 5-128849018901' 2024-09-16T17:28:47.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:47.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836530 2024-09-16T17:28:47.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:47.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:28:47.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836530 2024-09-16T17:28:47.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:47.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836530 2024-09-16T17:28:47.966 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836530 2024-09-16T17:28:47.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-09-16T17:28:47.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:28:48.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836530 -lt 21474836530 2024-09-16T17:28:48.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:48.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:48.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673008 2024-09-16T17:28:48.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:28:48.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673008 2024-09-16T17:28:48.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:48.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673008 2024-09-16T17:28:48.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-09-16T17:28:48.406 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673008 2024-09-16T17:28:48.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:28:48.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673009 -lt 42949673008 2024-09-16T17:28:48.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:48.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509480 2024-09-16T17:28:48.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:48.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:28:48.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509480 2024-09-16T17:28:48.838 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:48.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509480 2024-09-16T17:28:48.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509480' 2024-09-16T17:28:48.840 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509480 2024-09-16T17:28:48.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:28:49.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509480 -lt 64424509480 2024-09-16T17:28:49.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:49.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345949 2024-09-16T17:28:49.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:49.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:28:49.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345949 2024-09-16T17:28:49.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:49.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345949 2024-09-16T17:28:49.269 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345949 2024-09-16T17:28:49.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345949' 2024-09-16T17:28:49.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:28:49.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345949 -lt 85899345949 2024-09-16T17:28:49.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:49.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:49.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182426 2024-09-16T17:28:49.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T17:28:49.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182426 2024-09-16T17:28:49.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:49.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182426 2024-09-16T17:28:49.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182426' 2024-09-16T17:28:49.694 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182426 2024-09-16T17:28:49.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T17:28:50.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182426 -lt 107374182426 2024-09-16T17:28:50.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:28:50.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018901 2024-09-16T17:28:50.121 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:28:50.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T17:28:50.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018901 2024-09-16T17:28:50.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:28:50.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018901 2024-09-16T17:28:50.126 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018901 2024-09-16T17:28:50.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018901' 2024-09-16T17:28:50.126 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T17:28:50.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018901 -lt 128849018901 2024-09-16T17:28:50.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:28:50.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:28:50.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:28:51.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T17:28:51.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:28:51.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:28:51.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:28:51.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:28:51.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:28:51.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:28:51.116 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:28:51.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T17:28:51.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:28:51.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:28:51.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:28:52.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T17:28:52.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:28:52.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:28:52.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:84: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 4 5 2024-09-16T17:28:52.875 INFO:tasks.workunit.client.0.smithi039.stderr:set 1.0 pg_upmap mapping to [1,4,5] 2024-09-16T17:28:52.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:87: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-09-16T17:28:54.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-09-16T17:28:54.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-09-16T17:28:54.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-09-16T17:28:55.094 INFO:tasks.workunit.client.0.smithi039.stdout: 2, 2024-09-16T17:28:55.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:93: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back KILL osd.2 2024-09-16T17:28:55.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:28:55.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:28:55.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:28:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:28:55.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:28:55.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:28:55.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:94: TEST_repeer_on_down_acting_member_coming_back: ceph osd down osd.2 2024-09-16T17:28:55.828 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-16T17:28:55.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:97: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-09-16T17:28:57.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-09-16T17:28:57.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-09-16T17:28:57.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:103: TEST_repeer_on_down_acting_member_coming_back: activate_osd td/repeer-on-acting-back 2 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repeer-on-acting-back 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repeer-on-acting-back/2 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-09-16T17:28:58.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-16T17:28:58.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:28:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:28:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:28:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:28:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T17:28:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T17:28:58.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repeer-on-acting-back/2 2024-09-16T17:28:58.037 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T17:28:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T17:28:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=71d2946d-ca01-48d9-9119-8252f81049fd --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.472214/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:28:58.038 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repeer-on-acting-back/2/whoami 2024-09-16T17:28:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T17:28:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T17:28:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T17:28:58.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T17:28:58.067 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:28:58.064+0000 7f894c7b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:28:58.069 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:28:58.064+0000 7f894c7b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:28:58.070 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:28:58.068+0000 7f894c7b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:28:58.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T17:28:58.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:28:58.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:28:58.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:28:58.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:28:58.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:28:58.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:28:58.565 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:28:58.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:28:58.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:28:58.629 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:28:58.624+0000 7f894c7b17c0 -1 Falling back to public interface 2024-09-16T17:28:58.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:28:59.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:00.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:00.016 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:29:00.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:29:00.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:00.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:29:00.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:01.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:01.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:01.492 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:29:01.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:29:01.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:01.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:29:02.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:02.953 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:02.948+0000 7f894c7b17c0 -1 osd.2 47 log_to_monitors true 2024-09-16T17:29:03.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:03.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:03.071 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:29:03.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:29:03.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:03.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:29:03.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:04.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:04.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:04.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:29:04.538 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:29:04.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:04.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:29:05.135 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 51 up_thru 0 down_at 48 last_clean_interval [15,47) [v2:127.0.0.1:6818/1607738767,v1:127.0.0.1:6819/1607738767] [v2:127.0.0.1:6820/1607738767,v1:127.0.0.1:6821/1607738767] exists,up 3c955f00-9762-4991-85b7-355da0a2effa 2024-09-16T17:29:05.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:29:05.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:29:05.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:29:05.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:104: TEST_repeer_on_down_acting_member_coming_back: wait_for_osd up 2 2024-09-16T17:29:05.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:29:05.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:29:05.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:29:05.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:29:05.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:05.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:29:05.138 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:29:05.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:05.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:29:05.571 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 51 up_thru 0 down_at 48 last_clean_interval [15,47) [v2:127.0.0.1:6818/1607738767,v1:127.0.0.1:6819/1607738767] [v2:127.0.0.1:6820/1607738767,v1:127.0.0.1:6821/1607738767] exists,up 3c955f00-9762-4991-85b7-355da0a2effa 2024-09-16T17:29:05.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:29:05.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:29:05.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:29:05.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:107: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-09-16T17:29:07.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-09-16T17:29:07.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-09-16T17:29:07.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-09-16T17:29:07.770 INFO:tasks.workunit.client.0.smithi039.stdout: 2 2024-09-16T17:29:07.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-16T17:29:07.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-09-16T17:29:07.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:29:07.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:29:07.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:29:07.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-16T17:29:07.772 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:29:07.772 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:29:07.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:29:07.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:29:07.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:29:07.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-16T17:29:07.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:29:07.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:29:07.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:29:07.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:29:07.833 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:29:08.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:29:08.266 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:29:08.266 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-16T17:29:08.267 INFO:tasks.workunit.client.0.smithi039.stderr:3 2024-09-16T17:29:08.267 INFO:tasks.workunit.client.0.smithi039.stderr:4 2024-09-16T17:29:08.267 INFO:tasks.workunit.client.0.smithi039.stderr:5' 2024-09-16T17:29:08.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:29:08.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:29:08.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:29:08.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836536 2024-09-16T17:29:08.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836536 2024-09-16T17:29:08.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836536' 2024-09-16T17:29:08.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:29:08.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:29:08.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673015 2024-09-16T17:29:08.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673015 2024-09-16T17:29:08.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836536 1-42949673015' 2024-09-16T17:29:08.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:29:08.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:29:09.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=219043332100 2024-09-16T17:29:09.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 219043332100 2024-09-16T17:29:09.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836536 1-42949673015 2-219043332100' 2024-09-16T17:29:09.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:29:09.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T17:29:09.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345955 2024-09-16T17:29:09.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345955 2024-09-16T17:29:09.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836536 1-42949673015 2-219043332100 3-85899345955' 2024-09-16T17:29:09.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:29:09.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T17:29:09.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182432 2024-09-16T17:29:09.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182432 2024-09-16T17:29:09.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836536 1-42949673015 2-219043332100 3-85899345955 4-107374182432' 2024-09-16T17:29:09.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:29:09.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T17:29:09.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018907 2024-09-16T17:29:09.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018907 2024-09-16T17:29:09.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836536 1-42949673015 2-219043332100 3-85899345955 4-107374182432 5-128849018907' 2024-09-16T17:29:09.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:29:09.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836536 2024-09-16T17:29:09.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:29:09.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:29:09.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836536 2024-09-16T17:29:09.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:29:09.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836536 2024-09-16T17:29:09.520 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836536 2024-09-16T17:29:09.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-09-16T17:29:09.521 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:29:09.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836534 -lt 21474836536 2024-09-16T17:29:09.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:29:10.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:29:10.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:29:11.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836537 -lt 21474836536 2024-09-16T17:29:11.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:29:11.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673015 2024-09-16T17:29:11.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:29:11.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:29:11.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673015 2024-09-16T17:29:11.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:29:11.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673015 2024-09-16T17:29:11.429 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949673015 2024-09-16T17:29:11.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673015' 2024-09-16T17:29:11.429 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:29:11.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673015 -lt 42949673015 2024-09-16T17:29:11.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:29:11.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-219043332100 2024-09-16T17:29:11.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:29:11.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:29:11.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:29:11.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-219043332100 2024-09-16T17:29:11.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=219043332100 2024-09-16T17:29:11.882 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 219043332100 2024-09-16T17:29:11.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 219043332100' 2024-09-16T17:29:11.883 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:29:12.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 219043332100 -lt 219043332100 2024-09-16T17:29:12.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:29:12.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345955 2024-09-16T17:29:12.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:29:12.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T17:29:12.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345955 2024-09-16T17:29:12.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:29:12.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345955 2024-09-16T17:29:12.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345955' 2024-09-16T17:29:12.488 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 85899345955 2024-09-16T17:29:12.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T17:29:12.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345956 -lt 85899345955 2024-09-16T17:29:12.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:29:12.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182432 2024-09-16T17:29:12.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:29:12.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T17:29:12.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182432 2024-09-16T17:29:12.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:29:12.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182432 2024-09-16T17:29:12.980 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.4 seq 107374182432 2024-09-16T17:29:12.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182432' 2024-09-16T17:29:12.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T17:29:13.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182433 -lt 107374182432 2024-09-16T17:29:13.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:29:13.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018907 2024-09-16T17:29:13.627 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:29:13.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T17:29:13.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018907 2024-09-16T17:29:13.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:29:13.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018907 2024-09-16T17:29:13.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018907' 2024-09-16T17:29:13.631 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.5 seq 128849018907 2024-09-16T17:29:13.631 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T17:29:14.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018908 -lt 128849018907 2024-09-16T17:29:14.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:29:14.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:29:14.115 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:29:14.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T17:29:14.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:29:14.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:29:14.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:29:14.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:29:14.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:29:14.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:29:14.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:29:15.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T17:29:15.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:29:15.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:29:15.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:29:15.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T17:29:15.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:29:15.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:29:15.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: find td/repeer-on-acting-back -name '*osd*log' 2024-09-16T17:29:15.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: grep -q 'Active: got notify from previous acting member.*, requesting pg_temp change' td/repeer-on-acting-back/osd.5.log td/repeer-on-acting-back/osd.3.log td/repeer-on-acting-back/osd.4.log td/repeer-on-acting-back/osd.2.log td/repeer-on-acting-back/osd.1.log td/repeer-on-acting-back/osd.0.log 2024-09-16T17:29:15.807 INFO:tasks.workunit.client.0.smithi039.stdout:success 2024-09-16T17:29:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:119: TEST_repeer_on_down_acting_member_coming_back: echo success 2024-09-16T17:29:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:121: TEST_repeer_on_down_acting_member_coming_back: delete_pool test 2024-09-16T17:29:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-16T17:29:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-16T17:29:16.519 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-16T17:29:16.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:122: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back 2024-09-16T17:29:16.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:29:16.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:29:16.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:29:16.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:29:16.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:29:21.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:29:21.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:42: run: teardown td/repeer-on-acting-back 2024-09-16T17:29:21.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-09-16T17:29:21.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:29:21.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-09-16T17:29:21.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:29:21.858 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:29:21.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:29:21.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:29:21.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:29:21.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:29:21.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:29:21.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:29:21.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:29:21.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:29:21.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:29:21.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:29:21.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:29:21.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:29:21.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:29:21.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:29:21.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:29:21.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:29:21.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-09-16T17:29:21.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:29:21.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:29:21.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:29:21.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.472214 2024-09-16T17:29:21.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:29:21.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:29:21.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T17:29:21.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/repeer-on-acting-back 0 2024-09-16T17:29:21.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-09-16T17:29:21.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T17:29:21.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-09-16T17:29:21.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:29:21.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:29:21.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:29:21.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:29:21.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:29:21.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:29:21.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:29:21.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:29:21.918 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:29:21.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:29:21.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:29:21.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:29:21.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:29:21.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:29:21.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:29:21.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:29:21.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:29:21.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T17:29:21.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-09-16T17:29:21.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:29:21.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:29:21.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.472214 2024-09-16T17:29:21.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.472214 2024-09-16T17:29:21.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:29:21.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:29:21.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T17:29:21.926 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T17:29:21.927 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T17:29:21.937 INFO:tasks.workunit:Running workunit osd/repro_long_log.sh... 2024-09-16T17:29:21.937 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/repro_long_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh 2024-09-16T17:29:21.988 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T17:29:21.993 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T17:29:21.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T17:29:21.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T17:29:21.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T17:29:21.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T17:29:21.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T17:29:21.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T17:29:21.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T17:29:21.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T17:29:21.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/repro-long-log 2024-09-16T17:29:21.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:23: run: local dir=td/repro-long-log 2024-09-16T17:29:21.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:24: run: shift 2024-09-16T17:29:21.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: export CEPH_MON=127.0.0.1:7100 2024-09-16T17:29:21.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: CEPH_MON=127.0.0.1:7100 2024-09-16T17:29:21.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:27: run: export CEPH_ARGS 2024-09-16T17:29:21.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: uuidgen 2024-09-16T17:29:21.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: CEPH_ARGS+='--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none ' 2024-09-16T17:29:21.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-09-16T17:29:21.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: set 2024-09-16T17:29:21.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T17:29:21.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: local 'funcs=TEST_repro_long_log1 2024-09-16T17:29:21.997 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_repro_long_log2 2024-09-16T17:29:21.997 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_trim_max_entries 2024-09-16T17:29:21.997 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_trim_max_entries_with_dups' 2024-09-16T17:29:21.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-16T17:29:21.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-16T17:29:21.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-16T17:29:21.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-16T17:29:21.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-16T17:29:21.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:29:21.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-16T17:29:21.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:29:21.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:29:21.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:29:21.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:29:21.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:29:21.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:29:21.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:29:21.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:29:21.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:29:22.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:29:22.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:29:22.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:29:22.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:29:22.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:29:22.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:29:22.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:29:22.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:29:22.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:29:22.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-16T17:29:22.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:29:22.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:29:22.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:29:22.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.487707 2024-09-16T17:29:22.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:29:22.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:29:22.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-16T17:29:22.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:29:22.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:29:22.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:29:22.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.487707 2024-09-16T17:29:22.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:29:22.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T17:29:22.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T17:29:22.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:29:22.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-16T17:29:22.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log1 td/repro-long-log 2024-09-16T17:29:22.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:97: TEST_repro_long_log1: local dir=td/repro-long-log 2024-09-16T17:29:22.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:99: TEST_repro_long_log1: setup_log_test td/repro-long-log 2024-09-16T17:29:22.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-16T17:29:22.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-16T17:29:22.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-16T17:29:22.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-16T17:29:22.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:29:22.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:29:22.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:29:22.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-16T17:29:22.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-16T17:29:22.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:29:22.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:29:22.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:29:22.394 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:29:22.394 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:29:22.394 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:29:22.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:29:22.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T17:29:22.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:29:22.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:29:22.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:29:22.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:29:22.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:29:22.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:29:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:29:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:29:22.607 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:29:22.608 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:29:22.608 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:29:22.608 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:29:22.608 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.487707/ceph-mon.a.asok 2024-09-16T17:29:22.608 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:29:22.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.487707/ceph-mon.a.asok config get fsid 2024-09-16T17:29:22.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:29:22.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:29:22.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:29:22.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:29:22.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:29:22.727 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:29:22.728 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:29:22.728 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:29:22.729 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:29:22.729 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:29:22.729 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:29:22.729 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.487707/ceph-mon.a.asok 2024-09-16T17:29:22.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:29:22.729 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.487707/ceph-mon.a.asok config get mon_host 2024-09-16T17:29:22.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-16T17:29:22.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-16T17:29:22.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:29:22.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:29:22.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:29:22.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-16T17:29:22.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:29:23.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:29:23.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:29:23.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:29:23.755 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:29:23.755 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:29:23.755 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:29:23.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:29:23.755 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:29:23.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:29:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-16T17:29:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:29:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:29:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:29:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:29:23.794 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:29:23.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-16T17:29:23.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:29:23.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd7fa8a7-c1d8-480e-8d5b-b5b6be1cc9a2 2024-09-16T17:29:23.796 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 fd7fa8a7-c1d8-480e-8d5b-b5b6be1cc9a2 2024-09-16T17:29:23.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fd7fa8a7-c1d8-480e-8d5b-b5b6be1cc9a2' 2024-09-16T17:29:23.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:29:23.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzauhmWqBTMBAArHDZ+b29rghaxv8uSxNijQ== 2024-09-16T17:29:23.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzauhmWqBTMBAArHDZ+b29rghaxv8uSxNijQ=="}' 2024-09-16T17:29:23.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd7fa8a7-c1d8-480e-8d5b-b5b6be1cc9a2 -i td/repro-long-log/0/new.json 2024-09-16T17:29:24.107 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:29:24.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-16T17:29:24.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzauhmWqBTMBAArHDZ+b29rghaxv8uSxNijQ== --osd-uuid fd7fa8a7-c1d8-480e-8d5b-b5b6be1cc9a2 2024-09-16T17:29:24.156 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:24.152+0000 7fe287c247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:29:24.159 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:24.156+0000 7fe287c247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:29:24.160 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:24.156+0000 7fe287c247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:29:24.160 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:24.156+0000 7fe287c247c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:29:24.160 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:24.156+0000 7fe287c247c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:29:24.160 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:24.156+0000 7fe287c247c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:29:24.191 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:24.188+0000 7fe287c247c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-16T17:29:30.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-16T17:29:30.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:29:30.017 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:29:30.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:29:30.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:29:30.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:29:30.581 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:29:30.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:29:30.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:29:30.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:29:30.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:29:30.611 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:30.608+0000 7f64004da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:29:30.631 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:30.628+0000 7f64004da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:29:30.632 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:30.628+0000 7f64004da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:29:31.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:29:31.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:29:31.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:29:31.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:29:31.055 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:29:31.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:29:31.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:31.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:29:31.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:31.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:31.212 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:31.208+0000 7f64004da7c0 -1 Falling back to public interface 2024-09-16T17:29:31.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:32.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:32.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:32.530 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:29:32.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:29:32.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:32.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:32.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:33.714 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:33.708+0000 7f64004da7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:29:33.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:33.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:33.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:29:33.980 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:29:33.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:33.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:34.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:35.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:35.415 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:29:35.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:35.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:29:35.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:35.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:35.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:36.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:36.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:36.856 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:29:36.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:29:36.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:36.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:37.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:38.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:38.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:38.454 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:29:38.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:29:38.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:38.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:38.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:39.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:39.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:39.882 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:29:39.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:29:39.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:39.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:40.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:41.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:41.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:41.415 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:29:41.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:29:41.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:41.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:41.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:42.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:42.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:42.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:29:42.864 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:29:42.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:42.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:43.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:44.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:44.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:44.378 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:29:44.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:29:44.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:44.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:44.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:45.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:45.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:45.811 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:29:45.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:29:45.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:45.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:46.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:47.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:47.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:47.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:29:47.541 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:29:47.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:47.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:48.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:49.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:49.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:49.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:29:49.526 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:29:49.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:49.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:50.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:51.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:51.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:51.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:29:51.295 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:29:51.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:51.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:52.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:53.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:53.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:53.352 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:29:53.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:29:53.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:53.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:54.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:55.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:55.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:55.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:29:55.344 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:29:55.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:55.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:55.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:29:56.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:29:56.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:29:56.937 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:29:56.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:29:56.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:29:56.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:29:57.362 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3227568310,v1:127.0.0.1:6803/3227568310] [v2:127.0.0.1:6804/3227568310,v1:127.0.0.1:6805/3227568310] exists,up fd7fa8a7-c1d8-480e-8d5b-b5b6be1cc9a2 2024-09-16T17:29:57.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:29:57.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:29:57.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:29:57.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-16T17:29:57.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:29:57.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:29:57.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T17:29:57.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:29:57.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-16T17:29:57.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:29:57.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:29:57.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:29:57.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:29:57.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-16T17:29:57.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-16T17:29:57.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:29:57.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:29:57.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:29:57.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:29:57.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:29:57.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:29:57.368 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:29:57.368 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:29:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:29:57.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:29:57.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:29:57.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-16T17:29:57.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:29:57.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0865faa7-7b2f-4a38-848b-1903b749ab81 2024-09-16T17:29:57.371 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 0865faa7-7b2f-4a38-848b-1903b749ab81 2024-09-16T17:29:57.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0865faa7-7b2f-4a38-848b-1903b749ab81' 2024-09-16T17:29:57.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:29:57.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVa+hmBijTFhAAqojiiQp1x215zoQyloTQig== 2024-09-16T17:29:57.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVa+hmBijTFhAAqojiiQp1x215zoQyloTQig=="}' 2024-09-16T17:29:57.384 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0865faa7-7b2f-4a38-848b-1903b749ab81 -i td/repro-long-log/1/new.json 2024-09-16T17:29:57.914 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:29:57.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-16T17:29:57.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVa+hmBijTFhAAqojiiQp1x215zoQyloTQig== --osd-uuid 0865faa7-7b2f-4a38-848b-1903b749ab81 2024-09-16T17:29:57.960 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:57.955+0000 7f8fb8c517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:29:57.962 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:57.955+0000 7f8fb8c517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:29:57.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:57.959+0000 7f8fb8c517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:29:57.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:57.959+0000 7f8fb8c517c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:29:57.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:57.959+0000 7f8fb8c517c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:29:57.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:57.959+0000 7f8fb8c517c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:29:58.010 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:29:58.007+0000 7f8fb8c517c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-16T17:30:04.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-16T17:30:04.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:30:04.094 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T17:30:04.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T17:30:04.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:30:04.666 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:30:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T17:30:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:30:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:30:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:30:04.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:30:04.697 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:04.691+0000 7f5f0275a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:30:04.708 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:04.703+0000 7f5f0275a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:30:04.709 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:04.703+0000 7f5f0275a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:30:05.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T17:30:05.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:30:05.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:30:05.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:30:05.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:30:05.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:05.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:30:05.088 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:05.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:05.288 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:05.283+0000 7f5f0275a7c0 -1 Falling back to public interface 2024-09-16T17:30:05.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:06.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:06.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:06.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:30:06.576 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:30:06.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:06.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:07.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:07.776 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:07.771+0000 7f5f0275a7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T17:30:08.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:08.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:08.007 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:30:08.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:30:08.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:08.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:08.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:09.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:09.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:09.588 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:30:09.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:30:09.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:09.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:10.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:11.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:11.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:11.009 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:30:11.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:30:11.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:11.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:11.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:12.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:12.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:12.568 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:30:12.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:30:12.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:12.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:13.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:14.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:14.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:14.120 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:30:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:30:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:14.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:14.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:15.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:15.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:15.576 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:30:15.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:30:15.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:15.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:16.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:17.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:17.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:17.013 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:30:17.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:30:17.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:17.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:17.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:18.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:18.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:18.563 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:30:18.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:30:18.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:18.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:19.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:20.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:20.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:20.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:30:20.094 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:30:20.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:20.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:20.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:21.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:21.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:21.666 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:30:21.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:30:21.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:21.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:22.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:23.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:23.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:23.332 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:30:23.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:30:23.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:23.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:24.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:25.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:25.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:25.286 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:30:25.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:30:25.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:25.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:26.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:27.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:27.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:27.021 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:30:27.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:30:27.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:27.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:27.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:28.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:28.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:28.871 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:30:28.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:30:28.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:28.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:29.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:30.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:30.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:30.639 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:30:30.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:30:30.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:30.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:30:31.069 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/504898134,v1:127.0.0.1:6811/504898134] [v2:127.0.0.1:6812/504898134,v1:127.0.0.1:6813/504898134] exists,up 0865faa7-7b2f-4a38-848b-1903b749ab81 2024-09-16T17:30:31.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:30:31.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:30:31.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:30:31.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-16T17:30:31.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:30:31.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:30:31.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T17:30:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:30:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-16T17:30:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:30:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:30:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:30:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:30:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-16T17:30:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-16T17:30:31.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:30:31.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:30:31.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:30:31.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:30:31.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:30:31.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:30:31.077 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:30:31.077 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:30:31.077 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:30:31.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:30:31.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:30:31.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:30:31.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:30:31.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:30:31.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:30:31.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:30:31.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:30:31.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:30:31.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:30:31.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:30:31.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:30:31.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:30:31.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-16T17:30:31.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:30:31.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6169da95-0c0e-4fae-8baa-ad8f2608adce 2024-09-16T17:30:31.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6169da95-0c0e-4fae-8baa-ad8f2608adce' 2024-09-16T17:30:31.080 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 6169da95-0c0e-4fae-8baa-ad8f2608adce 2024-09-16T17:30:31.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:30:31.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3a+hmDnujBRAAR3XqE8/NJB98JvTAlckWYw== 2024-09-16T17:30:31.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3a+hmDnujBRAAR3XqE8/NJB98JvTAlckWYw=="}' 2024-09-16T17:30:31.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6169da95-0c0e-4fae-8baa-ad8f2608adce -i td/repro-long-log/2/new.json 2024-09-16T17:30:31.704 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:30:31.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-16T17:30:31.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3a+hmDnujBRAAR3XqE8/NJB98JvTAlckWYw== --osd-uuid 6169da95-0c0e-4fae-8baa-ad8f2608adce 2024-09-16T17:30:31.758 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:31.759+0000 7fa1f8e337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:30:31.761 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:31.759+0000 7fa1f8e337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:30:31.762 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:31.759+0000 7fa1f8e337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:30:31.763 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:31.763+0000 7fa1f8e337c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:30:31.763 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:31.763+0000 7fa1f8e337c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:30:31.763 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:31.763+0000 7fa1f8e337c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:30:31.826 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:31.823+0000 7fa1f8e337c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-16T17:30:37.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-16T17:30:37.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:30:37.423 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T17:30:37.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T17:30:37.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:30:37.982 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T17:30:37.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T17:30:37.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:30:37.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:30:37.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:30:37.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:30:38.012 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:38.010+0000 7fd3191287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:30:38.041 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:38.038+0000 7fd3191287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:30:38.043 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:38.042+0000 7fd3191287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:30:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T17:30:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:30:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:30:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:30:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:30:38.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:38.419 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:30:38.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:30:38.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:38.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:38.616 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:38.614+0000 7fd3191287c0 -1 Falling back to public interface 2024-09-16T17:30:38.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:39.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:39.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:39.849 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:30:39.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:30:39.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:39.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:40.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:41.202 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:30:41.198+0000 7fd3191287c0 -1 osd.2 0 log_to_monitors true 2024-09-16T17:30:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:41.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:30:41.277 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:30:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:41.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:41.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:42.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:42.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:42.813 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:30:42.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:30:42.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:42.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:43.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:44.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:44.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:44.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:30:44.291 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:30:44.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:44.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:44.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:45.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:45.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:45.918 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:30:45.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:30:45.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:45.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:46.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:47.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:47.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:47.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:30:47.392 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:30:47.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:47.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:47.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:48.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:48.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:48.962 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:30:48.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:30:48.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:48.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:49.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:50.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:50.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:50.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:30:50.518 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:30:50.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:50.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:51.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:52.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:52.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:52.003 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:30:52.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:30:52.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:52.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:52.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:53.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:53.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:53.434 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:30:53.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:30:53.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:53.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:54.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:55.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:55.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:55.205 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:30:55.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:30:55.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:55.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:56.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:57.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:57.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:57.514 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:30:57.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:30:57.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:57.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:30:58.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:30:59.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:30:59.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:30:59.478 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:30:59.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:30:59.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:30:59.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:31:00.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:01.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:01.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:01.226 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:31:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:31:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:31:02.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:03.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:03.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:03.201 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:31:03.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:31:03.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:03.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:31:03.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:04.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:04.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:04.639 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:31:04.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:31:04.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:04.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:31:05.085 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3339073106,v1:127.0.0.1:6819/3339073106] [v2:127.0.0.1:6820/3339073106,v1:127.0.0.1:6821/3339073106] exists,up 6169da95-0c0e-4fae-8baa-ad8f2608adce 2024-09-16T17:31:05.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:31:05.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:31:05.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:31:05.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-16T17:31:05.681 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T17:31:05.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-16T17:31:05.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-16T17:31:06.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-16T17:31:06.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-16T17:31:06.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-16T17:31:06.398 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:31:06.426 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:31:06.460 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:31:06.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-16T17:31:06.639 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:31:06.667 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:31:06.694 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:31:06.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-16T17:31:06.856 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:31:06.882 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:31:06.913 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:31:06.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-16T17:31:07.109 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:31:07.147 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:31:07.187 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:31:07.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-16T17:31:07.356 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:31:07.383 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:31:07.411 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:31:07.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-16T17:31:07.428 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-16T17:31:07.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:07.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:07.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:07.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:07.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:07.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:08.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:08.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:08.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:08.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:08.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:08.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:08.690 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:08.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:08.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:08.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:08.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:08.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:09.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:09.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:09.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:09.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:09.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:09.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:09.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:09.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:09.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:09.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:09.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:09.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:09.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:09.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:09.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:09.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:10.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:10.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:10.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:10.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:10.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:10.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:10.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-16T17:31:10.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:31:10.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-16T17:31:10.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:31:10.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:31:10.516 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836498 2024-09-16T17:31:10.540 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672974 2024-09-16T17:31:10.564 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509448 2024-09-16T17:31:10.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:31:13.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:13.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:31:13.749 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:31:13.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:13.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:31:13.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-16T17:31:13.913 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:31:13.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:31:13.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:14.077 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:31:14.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:14.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:31:14.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:31:14.235 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:31:14.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-16T17:31:14.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-16T17:31:14.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:14.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:14.464 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:14.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:14.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:14.590 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:14.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:14.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:14.716 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:14.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:14.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:14.816 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:14.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:14.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:14.925 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:14.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:14.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:15.060 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:15.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:15.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:15.168 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:15.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:15.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:15.277 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:15.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:15.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:15.403 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:15.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:15.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:15.554 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:15.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:15.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:15.671 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:15.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:15.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:15.780 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:15.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:15.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:15.889 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:15.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:15.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:15.997 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:16.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:16.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:16.123 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:16.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:16.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:16.240 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:16.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:16.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:16.341 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:16.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:16.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:16.441 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:16.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:16.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:16.558 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:16.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:31:16.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:31:16.667 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:31:16.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-16T17:31:16.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:31:16.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-16T17:31:16.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:31:16.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:31:16.823 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836502 2024-09-16T17:31:16.846 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672977 2024-09-16T17:31:16.871 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509451 2024-09-16T17:31:16.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:31:19.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:19.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:31:20.057 INFO:tasks.workunit.client.0.smithi039.stdout:21 2024-09-16T17:31:20.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:20.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:31:20.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-16T17:31:20.218 INFO:tasks.workunit.client.0.smithi039.stdout:21 2024-09-16T17:31:20.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:20.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:31:20.373 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:31:20.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:31:20.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:20.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:31:20.531 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:31:20.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:101: TEST_repro_long_log1: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:31:20.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:102: TEST_repro_long_log1: test_log_size 1.0 22 2024-09-16T17:31:20.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:31:20.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=22 2024-09-16T17:31:20.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:31:20.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:31:20.847 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836504 2024-09-16T17:31:20.871 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672980 2024-09-16T17:31:20.895 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509454 2024-09-16T17:31:20.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:31:23.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:23.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:31:24.083 INFO:tasks.workunit.client.0.smithi039.stdout:22 2024-09-16T17:31:24.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:24.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:31:24.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 22 2024-09-16T17:31:24.245 INFO:tasks.workunit.client.0.smithi039.stdout:22 2024-09-16T17:31:24.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:24.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:31:24.399 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:31:24.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:31:24.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:31:24.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:31:24.554 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:31:24.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-16T17:31:24.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-16T17:31:24.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:31:24.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-16T17:31:24.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:31:24.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:31:24.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:31:24.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:31:24.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:31:24.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:31:24.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:31:24.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:31:24.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:31:24.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:31:24.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:31:24.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:31:24.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:31:24.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:31:24.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:31:24.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:31:24.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:31:24.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:31:24.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-16T17:31:24.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:31:24.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:31:24.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:31:24.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.487707 2024-09-16T17:31:24.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:31:24.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:31:24.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-16T17:31:24.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-16T17:31:24.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-16T17:31:24.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-16T17:31:24.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-16T17:31:24.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:31:24.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-16T17:31:24.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:31:24.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:31:24.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:31:24.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:31:24.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:31:24.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:31:24.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:31:24.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:31:24.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:31:24.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:31:24.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:31:24.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:31:24.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:31:24.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:31:24.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:31:24.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:31:24.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:31:24.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:31:24.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-16T17:31:24.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:31:24.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:31:24.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:31:24.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.487707 2024-09-16T17:31:24.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:31:24.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:31:24.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-16T17:31:24.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:31:24.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:31:24.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:31:24.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.487707 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log2 td/repro-long-log 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:107: TEST_repro_long_log2: local dir=td/repro-long-log 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:109: TEST_repro_long_log2: setup_log_test td/repro-long-log 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:31:24.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:31:24.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-16T17:31:24.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-16T17:31:25.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:31:25.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:31:25.298 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:31:25.298 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:31:25.298 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:31:25.299 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:31:25.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:31:25.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T17:31:25.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:31:25.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:31:25.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:31:25.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:31:25.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:31:25.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:31:25.604 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:31:25.604 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:31:25.604 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:31:25.604 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:31:25.605 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:31:25.605 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:31:25.605 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.487707/ceph-mon.a.asok 2024-09-16T17:31:25.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:31:25.605 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.487707/ceph-mon.a.asok config get fsid 2024-09-16T17:31:25.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:31:25.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:31:25.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:31:25.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:31:25.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:31:25.726 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:31:25.726 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:31:25.727 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:31:25.727 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:31:25.727 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:31:25.727 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:31:25.727 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.487707/ceph-mon.a.asok 2024-09-16T17:31:25.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:31:25.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.487707/ceph-mon.a.asok config get mon_host 2024-09-16T17:31:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-16T17:31:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-16T17:31:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:31:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:31:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:31:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-16T17:31:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:31:26.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:31:26.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:31:26.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:31:26.365 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:31:26.365 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:31:26.365 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:31:26.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:31:26.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:31:26.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:31:26.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-16T17:31:26.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:31:26.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:31:26.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:31:26.403 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:31:26.404 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:31:26.404 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:31:26.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:31:26.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:31:26.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:31:26.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:31:26.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:31:26.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=069838e6-3c60-4dab-93a0-582dfd02e0f9 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 069838e6-3c60-4dab-93a0-582dfd02e0f9' 2024-09-16T17:31:26.406 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 069838e6-3c60-4dab-93a0-582dfd02e0f9 2024-09-16T17:31:26.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:31:26.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBua+hmwXUiGRAA2rDToumSoX3eY8EJ2cQURw== 2024-09-16T17:31:26.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBua+hmwXUiGRAA2rDToumSoX3eY8EJ2cQURw=="}' 2024-09-16T17:31:26.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 069838e6-3c60-4dab-93a0-582dfd02e0f9 -i td/repro-long-log/0/new.json 2024-09-16T17:31:26.799 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:31:26.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-16T17:31:26.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBua+hmwXUiGRAA2rDToumSoX3eY8EJ2cQURw== --osd-uuid 069838e6-3c60-4dab-93a0-582dfd02e0f9 2024-09-16T17:31:26.851 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:26.849+0000 7f15d82d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:31:26.853 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:26.849+0000 7f15d82d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:31:26.854 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:26.853+0000 7f15d82d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:31:26.855 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:26.853+0000 7f15d82d97c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:31:26.855 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:26.853+0000 7f15d82d97c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:31:26.855 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:26.853+0000 7f15d82d97c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:31:26.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:26.961+0000 7f15d82d97c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-16T17:31:32.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-16T17:31:32.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:31:32.744 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:31:32.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:31:32.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:31:33.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:31:33.323 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:31:33.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:31:33.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:31:33.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:31:33.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:31:33.354 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:33.353+0000 7f8d0aaff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:31:33.364 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:33.361+0000 7f8d0aaff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:31:33.366 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:33.361+0000 7f8d0aaff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:31:33.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:31:33.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:31:33.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:31:33.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:31:33.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:31:33.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:33.736 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:31:33.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:31:33.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:33.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:33.945 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:33.941+0000 7f8d0aaff7c0 -1 Falling back to public interface 2024-09-16T17:31:34.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:35.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:35.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:35.159 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:31:35.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:31:35.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:35.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:35.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:36.437 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:31:36.433+0000 7f8d0aaff7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:31:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:36.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:36.572 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:31:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:31:36.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:36.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:37.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:38.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:38.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:38.104 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:31:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:31:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:38.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:38.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:39.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:39.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:39.589 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:31:39.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:31:39.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:39.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:40.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:41.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:41.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:41.085 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:31:41.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:31:41.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:41.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:41.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:42.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:42.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:42.555 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:31:42.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:31:42.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:42.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:43.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:44.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:44.129 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:31:44.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:31:44.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:44.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:44.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:45.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:45.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:45.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:31:45.552 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:31:45.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:45.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:46.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:47.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:47.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:47.051 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:31:47.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:31:47.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:47.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:47.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:48.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:48.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:48.741 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:31:48.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:31:48.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:48.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:49.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:50.602 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:31:50.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:31:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:50.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:51.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:52.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:52.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:52.349 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:31:52.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:31:52.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:52.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:52.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:53.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:53.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:53.997 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:31:53.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:31:53.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:53.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:55.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:56.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:56.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:56.003 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:31:56.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:31:56.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:56.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:56.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:57.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:57.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:57.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:31:57.786 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:31:57.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:57.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:58.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:31:59.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:31:59.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:31:59.227 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:31:59.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:31:59.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:31:59.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:31:59.672 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3137225752,v1:127.0.0.1:6803/3137225752] [v2:127.0.0.1:6804/3137225752,v1:127.0.0.1:6805/3137225752] exists,up 069838e6-3c60-4dab-93a0-582dfd02e0f9 2024-09-16T17:31:59.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:31:59.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:31:59.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:31:59.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:31:59.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:31:59.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:31:59.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:31:59.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:31:59.675 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:31:59.675 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:31:59.675 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:31:59.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:31:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:31:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:31:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:31:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:31:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:31:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:31:59.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:31:59.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:31:59.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:31:59.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:31:59.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:31:59.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:31:59.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-16T17:31:59.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:31:59.677 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 42eba96e-90f1-45df-8d39-178039fbba95 2024-09-16T17:31:59.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=42eba96e-90f1-45df-8d39-178039fbba95 2024-09-16T17:31:59.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 42eba96e-90f1-45df-8d39-178039fbba95' 2024-09-16T17:31:59.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:31:59.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPa+hmI7+IKRAAd0I0OrUz9kBxDgwzGmyCWA== 2024-09-16T17:31:59.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPa+hmI7+IKRAAd0I0OrUz9kBxDgwzGmyCWA=="}' 2024-09-16T17:31:59.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 42eba96e-90f1-45df-8d39-178039fbba95 -i td/repro-long-log/1/new.json 2024-09-16T17:32:00.181 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:32:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-16T17:32:00.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCPa+hmI7+IKRAAd0I0OrUz9kBxDgwzGmyCWA== --osd-uuid 42eba96e-90f1-45df-8d39-178039fbba95 2024-09-16T17:32:00.230 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:00.225+0000 7f4af0dd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:00.249 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:00.245+0000 7f4af0dd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:00.251 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:00.249+0000 7f4af0dd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:00.251 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:00.249+0000 7f4af0dd97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:32:00.251 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:00.249+0000 7f4af0dd97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:32:00.251 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:00.249+0000 7f4af0dd97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:32:00.333 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:00.329+0000 7f4af0dd97c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-16T17:32:05.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-16T17:32:05.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:32:05.845 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T17:32:05.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T17:32:05.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:32:06.411 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:32:06.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T17:32:06.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:32:06.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:32:06.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:32:06.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:32:06.441 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:06.437+0000 7f994a0b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:06.473 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:06.469+0000 7f994a0b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:06.475 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:06.473+0000 7f994a0b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:06.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T17:32:06.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:32:06.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:32:06.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:32:06.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:32:06.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:06.869 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:32:06.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:32:06.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:06.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:07.056 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:07.053+0000 7f994a0b17c0 -1 Falling back to public interface 2024-09-16T17:32:07.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:08.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:08.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:08.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:32:08.306 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:32:08.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:08.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:08.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:09.684 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:09.681+0000 7f994a0b17c0 -1 osd.1 0 log_to_monitors true 2024-09-16T17:32:09.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:09.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:09.716 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:32:09.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:32:09.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:09.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:10.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:11.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:11.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:11.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:32:11.408 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:32:11.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:11.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:11.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:12.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:12.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:12.940 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:32:12.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:32:12.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:12.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:13.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:14.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:14.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:14.535 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:32:14.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:32:14.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:14.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:14.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:15.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:15.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:15.950 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:32:15.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:32:15.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:15.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:16.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:17.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:17.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:17.520 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:32:17.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:32:17.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:17.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:17.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:18.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:18.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:18.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:32:18.991 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:32:18.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:18.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:19.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:20.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:20.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:20.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:32:20.507 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:32:20.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:20.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:20.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:21.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:32:21.941 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:32:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:21.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:23.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:24.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:24.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:24.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:32:24.011 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:32:24.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:24.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:24.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:25.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:25.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:25.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:32:25.929 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:32:25.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:25.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:26.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:27.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:27.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:27.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:32:27.923 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:32:27.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:27.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:29.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:29.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:29.851 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:32:29.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:32:29.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:29.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:30.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:31.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:31.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:31.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:32:31.939 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:32:31.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:31.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:32:32.384 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3049415541,v1:127.0.0.1:6811/3049415541] [v2:127.0.0.1:6812/3049415541,v1:127.0.0.1:6813/3049415541] exists,up 42eba96e-90f1-45df-8d39-178039fbba95 2024-09-16T17:32:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:32:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:32:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:32:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-16T17:32:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:32:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:32:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T17:32:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:32:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-16T17:32:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:32:32.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:32:32.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:32:32.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:32:32.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:32:32.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:32:32.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:32:32.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:32:32.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:32:32.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:32:32.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:32:32.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:32:32.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:32:32.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-16T17:32:32.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:32:32.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=375d9def-46fd-4d57-8375-f6a7f7a3fb83 2024-09-16T17:32:32.391 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 375d9def-46fd-4d57-8375-f6a7f7a3fb83 2024-09-16T17:32:32.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 375d9def-46fd-4d57-8375-f6a7f7a3fb83' 2024-09-16T17:32:32.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:32:32.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCwa+hmPPp9GBAA6OJhjY9b9luELXZpWkCU7A== 2024-09-16T17:32:32.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCwa+hmPPp9GBAA6OJhjY9b9luELXZpWkCU7A=="}' 2024-09-16T17:32:32.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 375d9def-46fd-4d57-8375-f6a7f7a3fb83 -i td/repro-long-log/2/new.json 2024-09-16T17:32:33.007 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:32:33.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-16T17:32:33.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwa+hmPPp9GBAA6OJhjY9b9luELXZpWkCU7A== --osd-uuid 375d9def-46fd-4d57-8375-f6a7f7a3fb83 2024-09-16T17:32:33.064 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:33.060+0000 7fa007dd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:33.066 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:33.064+0000 7fa007dd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:33.068 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:33.064+0000 7fa007dd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:33.068 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:33.064+0000 7fa007dd67c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:32:33.068 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:33.064+0000 7fa007dd67c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:32:33.068 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:33.064+0000 7fa007dd67c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:32:33.130 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:33.124+0000 7fa007dd67c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-16T17:32:38.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-16T17:32:38.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:32:38.916 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T17:32:38.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T17:32:38.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:32:39.501 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T17:32:39.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T17:32:39.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:32:39.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:32:39.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:32:39.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:32:39.526 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:39.520+0000 7f639a66e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:39.528 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:39.524+0000 7f639a66e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:39.529 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:39.524+0000 7f639a66e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:32:39.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T17:32:39.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:32:39.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:32:39.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:32:39.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:32:39.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:39.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:32:39.960 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:32:39.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:39.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:40.069 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:40.064+0000 7f639a66e7c0 -1 Falling back to public interface 2024-09-16T17:32:40.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:41.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:41.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:41.382 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:32:41.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:32:41.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:41.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:41.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:32:42.544+0000 7f639a66e7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T17:32:42.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:42.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:42.909 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:32:42.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:32:42.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:42.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:43.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:44.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:44.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:44.331 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:32:44.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:32:44.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:44.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:44.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:45.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:45.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:45.759 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:32:45.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:32:45.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:45.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:46.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:47.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:47.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:47.302 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:32:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:32:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:47.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:47.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:48.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:48.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:48.931 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:32:48.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:32:48.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:48.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:49.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:50.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:50.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:50.422 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:32:50.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:32:50.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:50.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:50.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:51.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:51.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:51.853 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:32:51.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:32:51.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:51.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:52.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:53.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:53.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:53.327 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:32:53.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:32:53.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:53.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:53.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:54.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:54.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:54.947 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:32:54.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:32:54.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:54.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:55.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:56.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:56.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:56.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:32:56.836 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:32:56.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:56.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:57.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:32:58.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:32:58.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:32:58.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:32:58.531 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:32:58.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:32:58.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:32:59.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:00.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:00.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:00.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:33:00.596 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:33:00.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:00.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:33:01.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:02.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:02.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:02.449 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:33:02.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:33:02.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:02.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:33:03.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:04.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:04.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:04.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:33:04.474 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:33:04.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:04.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:33:05.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:06.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:06.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:06.258 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:33:06.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:33:06.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:06.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:33:06.681 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3309347358,v1:127.0.0.1:6819/3309347358] [v2:127.0.0.1:6820/3309347358,v1:127.0.0.1:6821/3309347358] exists,up 375d9def-46fd-4d57-8375-f6a7f7a3fb83 2024-09-16T17:33:06.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:33:06.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:33:06.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:33:06.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-16T17:33:07.272 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T17:33:07.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-16T17:33:07.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-16T17:33:07.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-16T17:33:07.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-16T17:33:07.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-16T17:33:07.904 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:33:07.933 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:33:07.966 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:33:07.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-16T17:33:08.131 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:33:08.161 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:33:08.190 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:33:08.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-16T17:33:08.355 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:33:08.385 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:33:08.413 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:33:08.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-16T17:33:08.575 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:33:08.603 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:33:08.631 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:33:08.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-16T17:33:08.793 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:33:08.823 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:33:08.850 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:33:08.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-16T17:33:08.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-16T17:33:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:09.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:09.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:09.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:09.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:10.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:10.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:10.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:10.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:10.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:10.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:10.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:10.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:10.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:10.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:10.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:10.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:10.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:10.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:11.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:11.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:11.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:11.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:11.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:11.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:11.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:11.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:11.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:11.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:11.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:11.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:11.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:11.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:11.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:11.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:12.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:12.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:12.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:12.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:33:12.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-16T17:33:12.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:33:12.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-16T17:33:12.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:33:12.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:33:12.474 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836502 2024-09-16T17:33:12.498 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672972 2024-09-16T17:33:12.522 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509453 2024-09-16T17:33:12.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:33:15.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:15.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:33:15.710 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:33:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:15.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:33:15.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-16T17:33:15.872 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:33:15.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:15.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:33:16.033 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:33:16.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:16.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:33:16.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:33:16.188 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:33:16.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-16T17:33:16.299 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-16T17:33:16.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:16.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:16.421 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:16.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:16.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:16.563 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:16.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:16.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:16.681 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:16.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:16.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:16.798 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:16.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:16.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:16.914 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:16.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:16.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:17.031 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:17.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:17.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:17.157 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:17.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:17.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:17.282 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:17.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:17.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:17.425 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:17.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:17.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:17.601 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:17.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:17.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:17.727 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:17.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:17.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:17.844 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:17.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:17.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:17.961 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:17.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:17.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:18.078 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:18.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:18.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:18.195 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:18.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:18.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:18.304 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:18.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:18.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:18.447 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:18.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:18.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:18.564 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:18.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:18.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:18.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:33:18.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:33:18.998 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:33:19.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-16T17:33:19.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:33:19.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-16T17:33:19.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:33:19.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:33:19.150 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836506 2024-09-16T17:33:19.174 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672975 2024-09-16T17:33:19.197 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509456 2024-09-16T17:33:19.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:33:22.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:22.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:33:22.386 INFO:tasks.workunit.client.0.smithi039.stdout:21 2024-09-16T17:33:22.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:33:22.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:22.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-16T17:33:22.545 INFO:tasks.workunit.client.0.smithi039.stdout:21 2024-09-16T17:33:22.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:33:22.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:22.701 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:33:22.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:22.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:33:22.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:33:22.854 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:33:22.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: jq .info.stats.up_primary 2024-09-16T17:33:22.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: ceph pg 1.0 query 2024-09-16T17:33:23.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: local PRIMARY=1 2024-09-16T17:33:23.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:111: TEST_repro_long_log2: kill_daemons td/repro-long-log TERM osd.1 2024-09-16T17:33:23.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:33:23.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:33:23.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:33:23.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:33:23.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:33:23.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:33:23.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: CEPH_ARGS='--osd-max-pg-log-entries=2 --osd-pg-log-dups-tracked=3 --no-mon-config' 2024-09-16T17:33:23.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: ceph-objectstore-tool --data-path td/repro-long-log/1 --pgid 1.0 --op trim-pg-log 2024-09-16T17:33:24.629 INFO:tasks.workunit.client.0.smithi039.stderr:Log bounds are: (19'20,19'41] 2024-09-16T17:33:24.630 INFO:tasks.workunit.client.0.smithi039.stdout:Removing keys 0000000019.00000000000000000021 - 0000000019.00000000000000000039 2024-09-16T17:33:24.658 INFO:tasks.workunit.client.0.smithi039.stderr:Finished trimming, now compacting... 2024-09-16T17:33:25.446 INFO:tasks.workunit.client.0.smithi039.stdout:Finished trimming pg log 2024-09-16T17:33:25.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:113: TEST_repro_long_log2: activate_osd td/repro-long-log 1 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repro-long-log 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repro-long-log/1 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T17:33:25.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:33:25.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T17:33:25.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:33:25.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:33:25.920 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:33:25.920 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:33:25.920 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:33:25.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:33:25.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:33:25.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:33:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:33:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:33:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:33:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:33:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:33:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:33:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T17:33:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T17:33:25.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repro-long-log/1 2024-09-16T17:33:25.922 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:33:25.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T17:33:25.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:33:25.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repro-long-log/1/whoami 2024-09-16T17:33:25.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T17:33:25.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T17:33:25.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T17:33:25.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T17:33:25.955 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:25.951+0000 7ffa58cef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:33:25.958 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:25.955+0000 7ffa58cef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:33:25.960 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:25.955+0000 7ffa58cef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:33:26.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T17:33:26.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:33:26.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:33:26.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:33:26.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:33:26.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:26.387 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:33:26.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:33:26.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:26.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:33:26.492 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:26.487+0000 7ffa58cef7c0 -1 Falling back to public interface 2024-09-16T17:33:26.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:27.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:33:27.825 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:33:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:27.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:33:28.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:28.822 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:28.819+0000 7ffa58cef7c0 -1 osd.1 19 log_to_monitors true 2024-09-16T17:33:29.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:29.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:29.244 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:33:29.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:33:29.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:29.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:33:29.761 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [10,19) [v2:127.0.0.1:6810/1626262053,v1:127.0.0.1:6811/1626262053] [v2:127.0.0.1:6812/1626262053,v1:127.0.0.1:6813/1626262053] exists,up 42eba96e-90f1-45df-8d39-178039fbba95 2024-09-16T17:33:29.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:33:29.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:33:29.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:33:29.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:114: TEST_repro_long_log2: wait_for_clean 2024-09-16T17:33:29.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T17:33:29.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T17:33:29.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T17:33:29.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T17:33:29.764 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T17:33:29.765 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T17:33:29.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T17:33:29.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T17:33:29.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T17:33:29.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T17:33:29.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T17:33:29.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T17:33:29.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T17:33:29.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T17:33:29.865 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T17:33:30.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T17:33:30.294 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-16T17:33:30.294 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-16T17:33:30.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T17:33:30.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:33:30.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T17:33:30.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836510 2024-09-16T17:33:30.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836510 2024-09-16T17:33:30.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510' 2024-09-16T17:33:30.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:33:30.496 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T17:33:30.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=98784247811 2024-09-16T17:33:30.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 98784247811 2024-09-16T17:33:30.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510 1-98784247811' 2024-09-16T17:33:30.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T17:33:30.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T17:33:30.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509460 2024-09-16T17:33:30.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509460 2024-09-16T17:33:30.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510 1-98784247811 2-64424509460' 2024-09-16T17:33:30.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:33:30.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836510 2024-09-16T17:33:30.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:33:30.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T17:33:30.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836510 2024-09-16T17:33:30.786 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:33:30.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836510 2024-09-16T17:33:30.787 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836510 2024-09-16T17:33:30.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-16T17:33:30.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:33:31.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836508 -lt 21474836510 2024-09-16T17:33:31.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T17:33:32.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T17:33:32.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T17:33:32.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-16T17:33:32.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:33:32.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-98784247811 2024-09-16T17:33:32.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:33:32.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T17:33:32.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-98784247811 2024-09-16T17:33:32.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:33:32.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=98784247811 2024-09-16T17:33:32.666 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 98784247811 2024-09-16T17:33:32.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 98784247811' 2024-09-16T17:33:32.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T17:33:33.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 98784247811 -lt 98784247811 2024-09-16T17:33:33.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T17:33:33.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509460 2024-09-16T17:33:33.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T17:33:33.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T17:33:33.105 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509460 2024-09-16T17:33:33.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T17:33:33.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509460 2024-09-16T17:33:33.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-16T17:33:33.107 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509460 2024-09-16T17:33:33.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T17:33:33.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509461 -lt 64424509460 2024-09-16T17:33:33.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T17:33:33.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:33:33.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:33:34.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-16T17:33:34.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T17:33:34.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T17:33:34.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T17:33:34.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T17:33:34.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T17:33:34.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T17:33:34.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T17:33:34.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-16T17:33:34.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T17:33:34.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T17:33:34.549 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T17:33:35.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-16T17:33:35.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T17:33:35.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T17:33:35.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:115: TEST_repro_long_log2: test_log_size 1.0 21 18 2024-09-16T17:33:35.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:33:35.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-16T17:33:35.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-09-16T17:33:35.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:33:35.238 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836513 2024-09-16T17:33:35.262 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 98784247814 2024-09-16T17:33:35.286 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509463 2024-09-16T17:33:35.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:33:38.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:38.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:33:38.472 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:33:38.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:38.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:33:38.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-16T17:33:38.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:38.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:33:38.788 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:33:38.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:33:38.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:33:38.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-09-16T17:33:38.949 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:33:38.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-16T17:33:38.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-16T17:33:38.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:33:38.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-16T17:33:38.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:33:38.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:33:38.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:33:38.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:33:38.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:33:39.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:33:39.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:33:39.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:33:39.061 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:33:39.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:33:39.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:33:39.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:33:39.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:33:39.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:33:39.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:33:39.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:33:39.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:33:39.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:33:39.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-16T17:33:39.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:33:39.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:33:39.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:33:39.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.487707 2024-09-16T17:33:39.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:33:39.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:33:39.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-16T17:33:39.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-16T17:33:39.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-16T17:33:39.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-16T17:33:39.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-16T17:33:39.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:33:39.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-16T17:33:39.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:33:39.086 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:33:39.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:33:39.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:33:39.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:33:39.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:33:39.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:33:39.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:33:39.088 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:33:39.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:33:39.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:33:39.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:33:39.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:33:39.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:33:39.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:33:39.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:33:39.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:33:39.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:33:39.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-16T17:33:39.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:33:39.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:33:39.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:33:39.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.487707 2024-09-16T17:33:39.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:33:39.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:33:39.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-16T17:33:39.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:33:39.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:33:39.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:33:39.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.487707 2024-09-16T17:33:39.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:33:39.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T17:33:39.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:33:39.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-16T17:33:39.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries td/repro-long-log 2024-09-16T17:33:39.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:120: TEST_trim_max_entries: local dir=td/repro-long-log 2024-09-16T17:33:39.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:122: TEST_trim_max_entries: setup_log_test td/repro-long-log 2024-09-16T17:33:39.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-16T17:33:39.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-16T17:33:39.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-16T17:33:39.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-16T17:33:39.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:33:39.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:33:39.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:33:39.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-16T17:33:39.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-16T17:33:39.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:33:39.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:33:39.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:33:39.453 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:33:39.453 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:33:39.453 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:33:39.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:33:39.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T17:33:39.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:33:39.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:33:39.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:33:39.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:33:39.665 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:33:39.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:33:39.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:33:39.666 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:33:39.667 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:33:39.667 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:33:39.667 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:33:39.667 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:33:39.667 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.487707/ceph-mon.a.asok 2024-09-16T17:33:39.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:33:39.667 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.487707/ceph-mon.a.asok config get fsid 2024-09-16T17:33:39.783 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:33:39.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:33:39.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:33:39.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:33:39.784 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:33:39.785 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:33:39.785 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:33:39.785 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:33:39.786 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:33:39.786 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:33:39.786 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:33:39.786 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.487707/ceph-mon.a.asok 2024-09-16T17:33:39.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:33:39.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.487707/ceph-mon.a.asok config get mon_host 2024-09-16T17:33:39.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-16T17:33:39.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-16T17:33:39.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:33:39.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:33:39.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:33:39.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-16T17:33:39.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:33:40.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:33:40.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:33:40.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:33:40.396 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:33:40.397 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:33:40.397 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:33:40.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:33:40.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:33:40.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:33:40.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-16T17:33:40.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:33:40.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:33:40.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:33:40.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:33:40.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-16T17:33:40.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:33:40.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:33:40.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:33:40.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:33:40.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-16T17:33:40.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-16T17:33:40.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:33:40.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:33:40.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:33:40.439 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:33:40.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:33:40.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:33:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:33:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:33:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:33:40.440 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:33:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:33:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:33:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:33:40.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:33:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:33:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:33:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:33:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:33:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:33:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:33:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:33:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:33:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-16T17:33:40.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:33:40.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=35ed3969-cb27-4a7c-87f4-c697c5d1aa5b 2024-09-16T17:33:40.442 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 35ed3969-cb27-4a7c-87f4-c697c5d1aa5b 2024-09-16T17:33:40.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 35ed3969-cb27-4a7c-87f4-c697c5d1aa5b' 2024-09-16T17:33:40.442 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:33:40.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0a+hmJsEhGxAAF5QSGP4Nq+B2H4vALa1Qag== 2024-09-16T17:33:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0a+hmJsEhGxAAF5QSGP4Nq+B2H4vALa1Qag=="}' 2024-09-16T17:33:40.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 35ed3969-cb27-4a7c-87f4-c697c5d1aa5b -i td/repro-long-log/0/new.json 2024-09-16T17:33:40.820 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:33:40.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-16T17:33:40.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0a+hmJsEhGxAAF5QSGP4Nq+B2H4vALa1Qag== --osd-uuid 35ed3969-cb27-4a7c-87f4-c697c5d1aa5b 2024-09-16T17:33:40.867 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:40.863+0000 7f70058307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:33:40.869 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:40.863+0000 7f70058307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:33:40.871 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:40.867+0000 7f70058307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:33:40.871 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:40.867+0000 7f70058307c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:33:40.872 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:40.867+0000 7f70058307c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:33:40.872 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:40.867+0000 7f70058307c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:33:40.959 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:40.955+0000 7f70058307c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-16T17:33:46.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-16T17:33:46.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:33:46.843 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:33:46.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:33:46.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:33:47.405 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:33:47.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:33:47.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:33:47.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:33:47.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:33:47.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:33:47.435 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:47.435+0000 7f316bd257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:33:47.438 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:47.435+0000 7f316bd257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:33:47.439 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:47.439+0000 7f316bd257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:33:47.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:33:47.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:33:47.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:33:47.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:33:47.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:33:47.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:47.855 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:33:47.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:33:47.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:47.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:33:48.020 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:48.019+0000 7f316bd257c0 -1 Falling back to public interface 2024-09-16T17:33:48.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:49.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:49.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:49.287 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:33:49.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:33:49.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:49.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:33:49.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:50.543 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:33:50.543+0000 7f316bd257c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:33:50.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:50.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:50.713 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:33:50.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:33:50.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:50.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:33:51.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:52.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:52.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:52.222 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:33:52.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:33:52.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:52.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:33:52.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:53.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:53.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:53.672 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:33:53.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:33:53.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:53.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:33:54.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:55.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:55.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:55.195 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:33:55.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:33:55.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:55.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:33:55.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:56.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:56.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:56.639 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:33:56.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:33:56.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:56.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:33:57.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:58.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:58.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:58.219 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:33:58.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:33:58.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:58.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:33:58.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:33:59.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:33:59.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:33:59.669 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:33:59.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:33:59.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:33:59.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:34:00.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:01.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:01.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:01.215 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:34:01.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:34:01.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:01.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:34:01.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:02.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:02.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:02.657 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:34:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:34:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:34:03.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:04.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:04.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:04.105 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:34:04.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:34:04.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:04.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:34:05.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:06.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:06.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:06.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:34:06.046 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:34:06.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:06.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:34:06.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:07.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:07.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:07.782 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:34:07.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:34:07.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:07.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:34:08.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:09.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:09.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:09.762 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:34:09.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:34:09.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:09.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:34:10.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:11.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:11.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:11.672 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:34:11.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:34:11.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:11.673 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:34:12.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:13.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:13.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:13.525 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:34:13.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:34:13.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:13.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:34:13.976 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3199247684,v1:127.0.0.1:6803/3199247684] [v2:127.0.0.1:6804/3199247684,v1:127.0.0.1:6805/3199247684] exists,up 35ed3969-cb27-4a7c-87f4-c697c5d1aa5b 2024-09-16T17:34:13.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:34:13.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:34:13.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:34:13.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-16T17:34:13.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:34:13.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:34:13.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T17:34:13.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:34:13.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-16T17:34:13.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:34:13.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:34:13.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:34:13.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:34:13.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-16T17:34:13.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-16T17:34:13.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:34:13.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:34:13.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:34:13.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:34:13.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:34:13.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:34:13.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:34:13.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:34:13.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:34:13.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:34:13.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:34:13.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-16T17:34:13.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:34:13.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b07b4422-c88a-44a1-a613-204432329ca7 2024-09-16T17:34:13.984 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 b07b4422-c88a-44a1-a613-204432329ca7 2024-09-16T17:34:13.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b07b4422-c88a-44a1-a613-204432329ca7' 2024-09-16T17:34:13.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:34:13.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVbOhmajx4OxAAGb7wjDqvJE7RU5ZaG3eU3Q== 2024-09-16T17:34:13.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVbOhmajx4OxAAGb7wjDqvJE7RU5ZaG3eU3Q=="}' 2024-09-16T17:34:13.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b07b4422-c88a-44a1-a613-204432329ca7 -i td/repro-long-log/1/new.json 2024-09-16T17:34:14.543 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:34:14.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-16T17:34:14.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVbOhmajx4OxAAGb7wjDqvJE7RU5ZaG3eU3Q== --osd-uuid b07b4422-c88a-44a1-a613-204432329ca7 2024-09-16T17:34:14.594 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:14.594+0000 7fb46d8a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:14.596 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:14.594+0000 7fb46d8a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:14.598 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:14.594+0000 7fb46d8a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:14.598 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:14.598+0000 7fb46d8a27c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:34:14.599 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:14.598+0000 7fb46d8a27c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:34:14.599 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:14.598+0000 7fb46d8a27c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:34:14.707 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:14.706+0000 7fb46d8a27c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-16T17:34:20.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-16T17:34:20.784 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:34:20.785 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T17:34:20.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T17:34:20.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:34:21.354 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:34:21.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T17:34:21.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:34:21.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:34:21.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:34:21.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:34:21.384 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:21.382+0000 7fbc2ae677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:21.396 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:21.394+0000 7fbc2ae677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:21.397 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:21.394+0000 7fbc2ae677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:21.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T17:34:21.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:34:21.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:34:21.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:34:21.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:34:21.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:21.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:34:21.786 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:34:21.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:21.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:21.976 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:21.974+0000 7fbc2ae677c0 -1 Falling back to public interface 2024-09-16T17:34:22.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:23.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:23.218 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:34:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:34:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:23.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:23.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:24.444 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:24.442+0000 7fbc2ae677c0 -1 osd.1 0 log_to_monitors true 2024-09-16T17:34:24.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:24.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:24.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:34:24.648 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:34:24.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:25.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:26.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:26.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:26.132 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:34:26.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:34:26.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:26.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:26.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:27.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:27.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:27.567 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:34:27.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:34:27.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:27.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:28.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:29.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:29.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:29.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:34:29.018 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:34:29.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:29.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:29.580 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:30.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:30.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:30.582 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:34:30.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:34:30.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:30.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:31.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:32.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:32.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:32.027 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:34:32.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:34:32.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:32.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:32.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:33.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:33.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:33.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:34:33.530 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:34:33.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:33.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:34.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:35.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:35.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:35.070 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:34:35.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:34:35.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:35.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:35.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:36.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:36.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:36.495 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:34:36.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:34:36.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:36.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:37.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:38.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:38.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:38.094 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:34:38.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:34:38.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:38.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:38.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:39.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:39.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:39.821 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:34:39.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:34:39.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:39.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:40.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:41.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:41.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:41.786 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:34:41.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:34:41.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:41.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:42.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:43.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:43.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:43.571 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:34:43.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:34:43.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:43.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:44.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:45.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:45.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:45.692 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:34:45.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:34:45.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:45.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:46.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:47.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:47.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:47.549 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:34:47.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:34:47.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:47.550 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:34:47.994 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1369254290,v1:127.0.0.1:6811/1369254290] [v2:127.0.0.1:6812/1369254290,v1:127.0.0.1:6813/1369254290] exists,up b07b4422-c88a-44a1-a613-204432329ca7 2024-09-16T17:34:47.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:34:47.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:34:47.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:34:47.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-16T17:34:47.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:34:47.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:34:47.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T17:34:47.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:34:48.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-16T17:34:48.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:34:48.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:34:48.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:34:48.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:34:48.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-16T17:34:48.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-16T17:34:48.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:34:48.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:34:48.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:34:48.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:34:48.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:34:48.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:34:48.001 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:34:48.001 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:34:48.001 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:34:48.001 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:34:48.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:34:48.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:34:48.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:34:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:34:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:34:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:34:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:34:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:34:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:34:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:34:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:34:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:34:48.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-16T17:34:48.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:34:48.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b090d907-c410-4848-b05e-69428e5daa0b 2024-09-16T17:34:48.003 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 b090d907-c410-4848-b05e-69428e5daa0b 2024-09-16T17:34:48.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b090d907-c410-4848-b05e-69428e5daa0b' 2024-09-16T17:34:48.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:34:48.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4bOhmR3kuARAAdlKRKSVGk+6hEAh6wJ1Kcw== 2024-09-16T17:34:48.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4bOhmR3kuARAAdlKRKSVGk+6hEAh6wJ1Kcw=="}' 2024-09-16T17:34:48.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b090d907-c410-4848-b05e-69428e5daa0b -i td/repro-long-log/2/new.json 2024-09-16T17:34:48.596 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:34:48.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-16T17:34:48.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4bOhmR3kuARAAdlKRKSVGk+6hEAh6wJ1Kcw== --osd-uuid b090d907-c410-4848-b05e-69428e5daa0b 2024-09-16T17:34:48.641 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:48.637+0000 7f2ead70b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:48.644 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:48.641+0000 7f2ead70b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:48.645 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:48.641+0000 7f2ead70b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:48.645 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:48.641+0000 7f2ead70b7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:34:48.646 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:48.641+0000 7f2ead70b7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:34:48.646 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:48.641+0000 7f2ead70b7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:34:48.742 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:48.741+0000 7f2ead70b7c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-16T17:34:54.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-16T17:34:54.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:34:54.647 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T17:34:54.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T17:34:54.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:34:55.188 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T17:34:55.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T17:34:55.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:34:55.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:34:55.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:34:55.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:34:55.218 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:55.217+0000 7f3fac0007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:55.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:55.217+0000 7f3fac0007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:55.221 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:55.217+0000 7f3fac0007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:34:55.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T17:34:55.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:34:55.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:34:55.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:34:55.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:34:55.613 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:34:55.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:55.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:34:55.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:34:55.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:55.775 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:55.773+0000 7f3fac0007c0 -1 Falling back to public interface 2024-09-16T17:34:56.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:57.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:57.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:57.042 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:34:57.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:34:57.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:57.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:34:57.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:34:58.267 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:34:58.265+0000 7f3fac0007c0 -1 osd.2 0 log_to_monitors true 2024-09-16T17:34:58.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:34:58.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:34:58.470 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:34:58.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:34:58.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:34:58.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:34:59.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:00.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:00.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:00.014 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:35:00.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:35:00.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:00.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:00.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:01.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:01.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:01.521 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:35:01.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:35:01.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:01.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:01.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:02.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:02.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:02.942 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:35:02.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:35:02.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:02.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:03.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:04.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:04.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:04.362 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:35:04.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:35:04.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:04.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:04.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:05.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:05.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:05.824 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:35:05.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:35:05.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:05.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:06.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:07.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:07.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:07.262 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:35:07.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:35:07.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:07.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:07.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:08.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:08.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:08.767 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:35:08.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:35:08.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:08.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:09.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:10.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:10.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:10.331 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:35:10.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:35:10.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:10.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:10.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:11.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:11.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:11.867 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:35:11.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:35:11.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:11.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:12.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:13.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:13.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:13.720 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:35:13.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:35:13.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:13.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:14.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:15.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:15.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:15.480 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:35:15.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:35:15.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:15.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:16.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:17.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:17.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:17.182 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:35:17.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:35:17.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:17.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:17.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:18.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:18.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:18.623 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:35:18.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:35:18.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:18.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:19.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:20.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:20.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:20.320 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:35:20.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:35:20.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:20.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:20.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:35:21.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:35:21.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:35:21.753 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-16T17:35:21.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T17:35:21.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:35:21.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:35:22.224 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3972936036,v1:127.0.0.1:6819/3972936036] [v2:127.0.0.1:6820/3972936036,v1:127.0.0.1:6821/3972936036] exists,up b090d907-c410-4848-b05e-69428e5daa0b 2024-09-16T17:35:22.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:35:22.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:35:22.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:35:22.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-16T17:35:22.827 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T17:35:22.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-16T17:35:22.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-16T17:35:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-16T17:35:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-16T17:35:23.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-16T17:35:23.411 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:35:23.439 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:35:23.470 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:35:23.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-16T17:35:23.671 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:35:23.710 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:35:23.749 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:35:23.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-16T17:35:23.972 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:35:24.000 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:35:24.036 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:35:24.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-16T17:35:24.198 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:35:24.230 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:35:24.259 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:35:24.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-16T17:35:24.459 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:35:24.486 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:35:24.516 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:35:24.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-16T17:35:24.534 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-16T17:35:24.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:24.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:24.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:25.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:25.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:25.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:25.342 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:25.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:25.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:25.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:25.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:25.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:25.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:25.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:25.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:26.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:26.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:26.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:26.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:26.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:26.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:26.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:26.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:26.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:26.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:26.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:26.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:26.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:26.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:26.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:26.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:27.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:27.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:27.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:27.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:27.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:27.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:27.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:27.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:35:27.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-16T17:35:27.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:35:27.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-16T17:35:27.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:35:27.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:35:27.677 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836498 2024-09-16T17:35:27.700 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672975 2024-09-16T17:35:27.725 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509448 2024-09-16T17:35:27.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:35:30.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:30.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:30.909 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:35:30.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:30.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:30.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-16T17:35:31.065 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:35:31.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:31.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:31.221 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:31.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:31.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:31.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:35:31.385 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:31.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-16T17:35:31.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-16T17:35:31.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:31.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:31.653 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:31.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:31.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:31.786 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:31.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:31.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:31.895 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:31.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:31.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:32.012 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:32.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:32.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:32.138 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:32.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:32.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:32.263 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:32.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:32.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:32.406 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:32.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:32.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:32.523 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:32.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:32.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:32.640 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:32.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:32.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:32.783 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:32.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:32.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:32.917 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:32.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:32.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:33.034 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:33.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:33.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:33.176 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:33.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:33.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:33.298 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:33.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:33.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:33.427 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:33.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:33.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:33.561 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:33.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:33.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:33.704 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:33.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:33.821 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:33.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:33.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:33.930 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:33.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:35:33.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:35:34.056 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:34.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-16T17:35:34.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:35:34.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-16T17:35:34.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:35:34.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:35:34.209 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836502 2024-09-16T17:35:34.233 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672979 2024-09-16T17:35:34.257 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509451 2024-09-16T17:35:34.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:35:37.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:37.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:37.445 INFO:tasks.workunit.client.0.smithi039.stdout:21 2024-09-16T17:35:37.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:37.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:37.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-16T17:35:37.603 INFO:tasks.workunit.client.0.smithi039.stdout:21 2024-09-16T17:35:37.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:37.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:37.757 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:37.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:35:37.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:37.923 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:37.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:124: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-09-16T17:35:38.058 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:35:38.085 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:35:38.114 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:35:38.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:125: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-09-16T17:35:38.275 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:35:38.304 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:35:38.331 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:35:38.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:126: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-09-16T17:35:38.491 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:35:38.519 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:35:38.547 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:35:38.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:127: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-09-16T17:35:38.700 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:35:38.727 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:35:38.755 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:35:38.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:128: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 0 2024-09-16T17:35:38.907 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:35:38.936 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:35:38.964 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:35:38.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:131: TEST_trim_max_entries: rados -p test rm foo 2024-09-16T17:35:39.112 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:39.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:132: TEST_trim_max_entries: test_log_size 1.0 18 2024-09-16T17:35:39.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:35:39.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-09-16T17:35:39.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:35:39.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:35:39.265 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836505 2024-09-16T17:35:39.289 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672982 2024-09-16T17:35:39.313 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509454 2024-09-16T17:35:39.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:35:42.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:42.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:42.502 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:35:42.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:42.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:42.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-09-16T17:35:42.662 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:35:42.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:42.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:42.817 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:42.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:42.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:42.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:35:42.980 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:42.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:133: TEST_trim_max_entries: rados -p test rm foo 2024-09-16T17:35:43.107 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:43.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:134: TEST_trim_max_entries: test_log_size 1.0 15 2024-09-16T17:35:43.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:35:43.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-09-16T17:35:43.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:35:43.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:35:43.254 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836507 2024-09-16T17:35:43.278 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672985 2024-09-16T17:35:43.302 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509457 2024-09-16T17:35:43.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:35:46.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:46.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:46.489 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:35:46.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:46.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:46.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-09-16T17:35:46.644 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:35:46.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:46.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:46.801 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:46.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:46.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:46.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:35:46.960 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:46.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:135: TEST_trim_max_entries: rados -p test rm foo 2024-09-16T17:35:47.076 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:47.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:136: TEST_trim_max_entries: test_log_size 1.0 12 2024-09-16T17:35:47.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:35:47.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-09-16T17:35:47.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:35:47.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:35:47.225 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836510 2024-09-16T17:35:47.248 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672987 2024-09-16T17:35:47.272 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509460 2024-09-16T17:35:47.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:35:50.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:50.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:50.457 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:35:50.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:50.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:50.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-09-16T17:35:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:35:50.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:50.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:50.768 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:50.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:50.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:50.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:35:50.925 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:50.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:137: TEST_trim_max_entries: rados -p test rm foo 2024-09-16T17:35:51.071 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:138: TEST_trim_max_entries: test_log_size 1.0 9 2024-09-16T17:35:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:35:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-09-16T17:35:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:35:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:35:51.220 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836513 2024-09-16T17:35:51.244 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672990 2024-09-16T17:35:51.267 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509462 2024-09-16T17:35:51.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:35:54.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:54.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:54.453 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:35:54.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:54.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:54.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-09-16T17:35:54.609 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:35:54.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:54.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:54.764 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:54.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:54.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:54.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:35:54.922 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:54.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:139: TEST_trim_max_entries: rados -p test rm foo 2024-09-16T17:35:55.049 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:140: TEST_trim_max_entries: test_log_size 1.0 6 2024-09-16T17:35:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:35:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-09-16T17:35:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:35:55.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:35:55.197 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836516 2024-09-16T17:35:55.222 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672993 2024-09-16T17:35:55.246 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509465 2024-09-16T17:35:55.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:35:58.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:58.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:58.430 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:35:58.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:58.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:35:58.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-09-16T17:35:58.590 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:35:58.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:58.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:58.746 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:58.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:35:58.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:35:58.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:35:58.909 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:35:58.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:141: TEST_trim_max_entries: rados -p test rm foo 2024-09-16T17:35:59.019 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:35:59.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:142: TEST_trim_max_entries: test_log_size 1.0 3 2024-09-16T17:35:59.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:35:59.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-16T17:35:59.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:35:59.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:35:59.170 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836519 2024-09-16T17:35:59.194 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672996 2024-09-16T17:35:59.219 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509468 2024-09-16T17:35:59.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:36:02.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:02.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:36:02.407 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:36:02.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:02.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:36:02.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-16T17:36:02.572 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:36:02.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:02.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:36:02.730 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:02.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:36:02.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:02.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:36:02.885 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:02.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:145: TEST_trim_max_entries: rados -p test rm foo 2024-09-16T17:36:03.039 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:36:03.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:146: TEST_trim_max_entries: test_log_size 1.0 4 2024-09-16T17:36:03.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:36:03.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-16T17:36:03.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:36:03.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:36:03.193 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836521 2024-09-16T17:36:03.217 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672999 2024-09-16T17:36:03.240 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509471 2024-09-16T17:36:03.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:36:06.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:06.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:36:06.427 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:36:06.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:36:06.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:06.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-16T17:36:06.590 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:36:06.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:06.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:36:06.745 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:06.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:06.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:36:06.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:36:06.901 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:06.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:147: TEST_trim_max_entries: rados -p test rm foo 2024-09-16T17:36:07.008 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:36:07.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:148: TEST_trim_max_entries: test_log_size 1.0 3 2024-09-16T17:36:07.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:36:07.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-16T17:36:07.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:36:07.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:36:07.159 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836524 2024-09-16T17:36:07.182 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949673001 2024-09-16T17:36:07.205 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509474 2024-09-16T17:36:07.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:36:10.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:10.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:36:10.389 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:36:10.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:10.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:36:10.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-16T17:36:10.546 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:36:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:10.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:36:10.696 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:10.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:10.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:36:10.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:36:10.846 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:10.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:149: TEST_trim_max_entries: rados -p test rm foo 2024-09-16T17:36:10.961 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:36:10.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:150: TEST_trim_max_entries: test_log_size 1.0 4 2024-09-16T17:36:10.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:36:10.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-16T17:36:10.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:36:10.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:36:11.109 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836527 2024-09-16T17:36:11.133 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949673004 2024-09-16T17:36:11.156 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509476 2024-09-16T17:36:11.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:36:14.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:14.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:36:14.342 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:36:14.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:14.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-16T17:36:14.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:36:14.498 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:36:14.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:14.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:36:14.655 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:14.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:14.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:36:14.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:36:14.815 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:14.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:151: TEST_trim_max_entries: rados -p test rm foo 2024-09-16T17:36:14.955 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:36:14.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:152: TEST_trim_max_entries: test_log_size 1.0 3 2024-09-16T17:36:14.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:36:14.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-16T17:36:14.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:36:14.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:36:15.105 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836530 2024-09-16T17:36:15.128 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949673007 2024-09-16T17:36:15.152 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509479 2024-09-16T17:36:15.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:36:18.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:18.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:36:18.338 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:36:18.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:18.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:36:18.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-16T17:36:18.497 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:36:18.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:36:18.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:18.661 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:18.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:36:18.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:36:18.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:36:18.825 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:18.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-16T17:36:18.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-16T17:36:18.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:36:18.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-16T17:36:18.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:36:18.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:36:18.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:36:18.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:36:18.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:36:19.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:36:19.138 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:36:19.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:36:19.140 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:36:19.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:36:19.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:36:19.142 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:36:19.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:36:19.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:36:19.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:36:19.144 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:36:19.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:36:19.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:36:19.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-16T17:36:19.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:36:19.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:36:19.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:36:19.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.487707 2024-09-16T17:36:19.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:36:19.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:36:19.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-16T17:36:19.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-16T17:36:19.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-16T17:36:19.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-16T17:36:19.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-16T17:36:19.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:36:19.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-16T17:36:19.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:36:19.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:36:19.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:36:19.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:36:19.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:36:19.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:36:19.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:36:19.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:36:19.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:36:19.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:36:19.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:36:19.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:36:19.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:36:19.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:36:19.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:36:19.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:36:19.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:36:19.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:36:19.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-16T17:36:19.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:36:19.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:36:19.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:36:19.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.487707 2024-09-16T17:36:19.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:36:19.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:36:19.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-16T17:36:19.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T17:36:19.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:36:19.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:36:19.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.487707 2024-09-16T17:36:19.189 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T17:36:19.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries_with_dups td/repro-long-log 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:157: TEST_trim_max_entries_with_dups: local dir=td/repro-long-log 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:159: TEST_trim_max_entries_with_dups: setup_log_test td/repro-long-log 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-16T17:36:19.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-16T17:36:19.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T17:36:19.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:36:19.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:36:19.569 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:36:19.569 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:36:19.569 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:36:19.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:36:19.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T17:36:19.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T17:36:19.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T17:36:19.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:36:19.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:36:19.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T17:36:19.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T17:36:19.773 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:36:19.773 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:36:19.774 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:36:19.774 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:36:19.774 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:36:19.774 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:36:19.774 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.487707/ceph-mon.a.asok 2024-09-16T17:36:19.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:36:19.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.487707/ceph-mon.a.asok config get fsid 2024-09-16T17:36:19.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T17:36:19.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T17:36:19.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T17:36:19.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T17:36:19.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T17:36:19.894 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T17:36:19.894 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T17:36:19.895 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T17:36:19.895 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T17:36:19.895 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:36:19.895 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:36:19.896 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.487707/ceph-mon.a.asok 2024-09-16T17:36:19.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T17:36:19.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.487707/ceph-mon.a.asok config get mon_host 2024-09-16T17:36:20.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-16T17:36:20.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-16T17:36:20.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T17:36:20.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T17:36:20.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T17:36:20.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-16T17:36:20.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T17:36:20.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T17:36:20.913 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:36:20.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:36:20.914 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:36:20.914 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:36:20.914 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:36:20.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:36:20.914 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:36:20.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T17:36:20.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-16T17:36:20.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:36:20.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:36:20.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:36:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:36:20.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-16T17:36:20.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:36:20.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bfd05740-b6a1-4c93-afad-1796af5bd69d 2024-09-16T17:36:20.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bfd05740-b6a1-4c93-afad-1796af5bd69d' 2024-09-16T17:36:20.954 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 bfd05740-b6a1-4c93-afad-1796af5bd69d 2024-09-16T17:36:20.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:36:20.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUbOhmZGq9ORAA77/6peyYXMssk4gAKy96Ow== 2024-09-16T17:36:20.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUbOhmZGq9ORAA77/6peyYXMssk4gAKy96Ow=="}' 2024-09-16T17:36:20.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bfd05740-b6a1-4c93-afad-1796af5bd69d -i td/repro-long-log/0/new.json 2024-09-16T17:36:21.283 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:21.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-16T17:36:21.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUbOhmZGq9ORAA77/6peyYXMssk4gAKy96Ow== --osd-uuid bfd05740-b6a1-4c93-afad-1796af5bd69d 2024-09-16T17:36:21.324 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:21.320+0000 7f62f78467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:36:21.326 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:21.324+0000 7f62f78467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:36:21.328 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:21.324+0000 7f62f78467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:36:21.328 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:21.324+0000 7f62f78467c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:36:21.328 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:21.324+0000 7f62f78467c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:36:21.328 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:21.324+0000 7f62f78467c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-16T17:36:21.359 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:21.356+0000 7f62f78467c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-16T17:36:27.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-16T17:36:27.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:36:27.221 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-16T17:36:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T17:36:27.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:36:27.792 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-16T17:36:27.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T17:36:27.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:36:27.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:36:27.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:36:27.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:36:27.824 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:27.819+0000 7fbad0f037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:36:27.834 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:27.827+0000 7fbad0f037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:36:27.835 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:27.831+0000 7fbad0f037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:36:28.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T17:36:28.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:36:28.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T17:36:28.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:36:28.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:36:28.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:28.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:36:28.210 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:36:28.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:28.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:28.393 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:28.387+0000 7fbad0f037c0 -1 Falling back to public interface 2024-09-16T17:36:28.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:29.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:29.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:29.623 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:36:29.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:36:29.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:29.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:30.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:30.902 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:30.895+0000 7fbad0f037c0 -1 osd.0 0 log_to_monitors true 2024-09-16T17:36:31.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:31.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:31.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:36:31.038 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:36:31.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:31.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:31.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:32.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:32.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:32.597 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:36:32.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:36:32.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:32.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:33.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:34.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:34.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:34.021 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:36:34.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:36:34.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:34.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:34.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:35.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:35.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:35.447 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:36:35.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:36:35.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:35.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:35.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:36.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:36.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:36.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:36:36.979 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:36:36.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:36.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:37.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:38.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:38.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:38.414 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:36:38.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:36:38.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:38.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:38.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:39.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:39.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:39.922 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:36:39.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:36:39.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:39.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:40.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:41.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:41.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:41.389 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:36:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:36:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:41.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:41.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:42.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:42.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:42.835 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:36:42.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:36:42.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:42.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:43.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:44.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:44.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:44.624 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:36:44.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:36:44.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:44.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:45.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:46.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:46.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:46.152 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:36:46.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:36:46.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:46.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:48.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:48.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:48.091 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:36:48.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:36:48.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:48.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:49.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:50.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:50.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:50.267 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:36:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:36:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:50.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:51.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:51.926 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:36:51.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:36:51.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:51.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:52.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:36:53.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:36:53.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:36:53.395 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:36:53.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:36:53.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:36:53.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T17:36:53.819 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3538775054,v1:127.0.0.1:6803/3538775054] [v2:127.0.0.1:6804/3538775054,v1:127.0.0.1:6805/3538775054] exists,up bfd05740-b6a1-4c93-afad-1796af5bd69d 2024-09-16T17:36:53.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:36:53.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:36:53.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:36:53.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-16T17:36:53.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:36:53.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:36:53.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T17:36:53.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:36:53.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-16T17:36:53.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:36:53.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:36:53.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:36:53.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:36:53.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-16T17:36:53.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-16T17:36:53.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:36:53.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:36:53.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:36:53.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:36:53.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:36:53.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:36:53.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:36:53.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:36:53.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:36:53.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:36:53.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:36:53.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:36:53.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:36:53.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:36:53.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:36:53.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:36:53.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:36:53.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:36:53.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:36:53.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:36:53.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:36:53.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:36:53.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-16T17:36:53.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:36:53.826 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 a97aee1a-2c6c-40da-806a-ab5970d4793f 2024-09-16T17:36:53.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a97aee1a-2c6c-40da-806a-ab5970d4793f 2024-09-16T17:36:53.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a97aee1a-2c6c-40da-806a-ab5970d4793f' 2024-09-16T17:36:53.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:36:53.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1bOhmHplWMhAA+AlSqk8bIq9uIXfXBj4Xvw== 2024-09-16T17:36:53.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1bOhmHplWMhAA+AlSqk8bIq9uIXfXBj4Xvw=="}' 2024-09-16T17:36:53.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a97aee1a-2c6c-40da-806a-ab5970d4793f -i td/repro-long-log/1/new.json 2024-09-16T17:36:54.514 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:36:54.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-16T17:36:54.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1bOhmHplWMhAA+AlSqk8bIq9uIXfXBj4Xvw== --osd-uuid a97aee1a-2c6c-40da-806a-ab5970d4793f 2024-09-16T17:36:54.565 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:54.559+0000 7ff6cc9427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:36:54.568 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:54.563+0000 7ff6cc9427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:36:54.569 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:54.563+0000 7ff6cc9427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:36:54.569 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:54.563+0000 7ff6cc9427c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:36:54.570 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:54.563+0000 7ff6cc9427c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:36:54.570 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:54.563+0000 7ff6cc9427c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-16T17:36:54.653 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:36:54.651+0000 7ff6cc9427c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-16T17:37:00.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-16T17:37:00.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:37:00.233 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-16T17:37:00.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T17:37:00.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:37:00.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T17:37:00.789 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-16T17:37:00.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:37:00.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:37:00.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:37:00.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:37:00.819 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:00.815+0000 7f399fcac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:37:00.826 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:00.823+0000 7f399fcac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:37:00.828 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:00.823+0000 7f399fcac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:37:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T17:37:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:37:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T17:37:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:37:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:37:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:01.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:37:01.228 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:37:01.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:01.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:01.387 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:01.387+0000 7f399fcac7c0 -1 Falling back to public interface 2024-09-16T17:37:01.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:02.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:02.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:02.744 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:37:02.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:37:02.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:02.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:03.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:03.901 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:03.899+0000 7f399fcac7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T17:37:04.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:04.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:04.163 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:37:04.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:37:04.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:04.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:04.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:05.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:05.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:05.608 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:37:05.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:37:05.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:05.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:06.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:07.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:07.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:07.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:37:07.278 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:37:07.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:07.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:07.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:08.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:08.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:08.771 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:37:08.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:37:08.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:08.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:09.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:10.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:10.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:10.200 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:37:10.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:37:10.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:10.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:10.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:11.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:11.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:11.764 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:37:11.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:37:11.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:11.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:12.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:13.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:13.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:13.293 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:37:13.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:37:13.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:13.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:13.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:14.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:14.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:14.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:37:14.813 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:37:14.813 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:14.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:15.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:16.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:16.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:16.259 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:37:16.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:37:16.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:16.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:17.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:18.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:18.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:18.261 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:37:18.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:37:18.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:18.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:19.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:20.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:20.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:20.633 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:37:20.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:37:20.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:20.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:21.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:22.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:22.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:22.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:37:22.656 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:37:22.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:22.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:23.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:24.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:24.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:24.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:37:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:37:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:25.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:26.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:26.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:26.539 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:37:26.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:37:26.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:26.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T17:37:26.992 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2005516982,v1:127.0.0.1:6811/2005516982] [v2:127.0.0.1:6812/2005516982,v1:127.0.0.1:6813/2005516982] exists,up a97aee1a-2c6c-40da-806a-ab5970d4793f 2024-09-16T17:37:26.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:37:26.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:37:26.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:37:26.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-16T17:37:26.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-16T17:37:26.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T17:37:26.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T17:37:26.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T17:37:26.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-16T17:37:26.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-16T17:37:26.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T17:37:26.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T17:37:26.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T17:37:26.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-16T17:37:26.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-16T17:37:26.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T17:37:26.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T17:37:26.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-16T17:37:26.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T17:37:26.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T17:37:26.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T17:37:26.999 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T17:37:26.999 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:37:26.999 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:37:26.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:37:26.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' 2024-09-16T17:37:26.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T17:37:26.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T17:37:26.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T17:37:26.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-16T17:37:27.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-16T17:37:27.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T17:37:27.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T17:37:27.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T17:37:27.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T17:37:27.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T17:37:27.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T17:37:27.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-16T17:37:27.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T17:37:27.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=99129ba7-e8b7-431e-bae5-b79dbfccef61 2024-09-16T17:37:27.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 99129ba7-e8b7-431e-bae5-b79dbfccef61' 2024-09-16T17:37:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 99129ba7-e8b7-431e-bae5-b79dbfccef61 2024-09-16T17:37:27.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T17:37:27.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXbOhmll0IARAAPwWDmgZQxHFHwvE38ibWMA== 2024-09-16T17:37:27.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXbOhmll0IARAAPwWDmgZQxHFHwvE38ibWMA=="}' 2024-09-16T17:37:27.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 99129ba7-e8b7-431e-bae5-b79dbfccef61 -i td/repro-long-log/2/new.json 2024-09-16T17:37:27.472 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:37:27.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-16T17:37:27.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXbOhmll0IARAAPwWDmgZQxHFHwvE38ibWMA== --osd-uuid 99129ba7-e8b7-431e-bae5-b79dbfccef61 2024-09-16T17:37:27.522 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:27.518+0000 7fba7f7587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:37:27.526 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:27.526+0000 7fba7f7587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:37:27.527 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:27.526+0000 7fba7f7587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:37:27.528 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:27.526+0000 7fba7f7587c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:37:27.528 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:27.526+0000 7fba7f7587c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:37:27.528 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:27.526+0000 7fba7f7587c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-16T17:37:27.566 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:27.566+0000 7fba7f7587c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-16T17:37:33.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-16T17:37:33.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T17:37:33.405 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-16T17:37:33.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T17:37:33.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T17:37:33.996 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-16T17:37:33.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T17:37:33.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=963e3a5e-1416-4378-8242-acb50b6a5d4b --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.487707/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T17:37:33.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T17:37:33.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T17:37:33.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T17:37:34.027 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:34.026+0000 7fd6084507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:37:34.039 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:34.038+0000 7fd6084507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:37:34.041 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:34.038+0000 7fd6084507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T17:37:34.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T17:37:34.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T17:37:34.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T17:37:34.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T17:37:34.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T17:37:34.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:34.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T17:37:34.437 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:37:34.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:34.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:34.588 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:34.586+0000 7fd6084507c0 -1 Falling back to public interface 2024-09-16T17:37:34.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:35.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:35.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:35.898 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-16T17:37:35.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T17:37:35.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:35.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:36.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:37.049 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-16T17:37:37.046+0000 7fd6084507c0 -1 osd.2 0 log_to_monitors true 2024-09-16T17:37:37.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:37.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:37.361 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:37:37.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T17:37:37.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:37.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:37.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:38.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:38.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:38.852 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:37:38.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T17:37:38.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:38.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:39.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:40.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:40.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:40.286 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:37:40.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T17:37:40.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:40.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:40.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:41.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:41.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:41.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T17:37:41.839 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-16T17:37:41.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:41.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:42.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:43.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:43.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:43.395 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:37:43.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T17:37:43.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:43.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:43.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:44.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:44.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:44.901 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-16T17:37:44.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T17:37:44.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:44.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:45.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:46.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:46.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:46.317 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-16T17:37:46.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T17:37:46.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:46.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:46.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:47.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:47.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:47.844 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:37:47.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T17:37:47.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:47.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:48.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:49.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:49.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:49.314 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:37:49.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T17:37:49.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:49.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:49.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:50.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:50.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:50.881 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-16T17:37:50.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T17:37:50.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:50.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:51.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:52.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:52.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:52.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T17:37:52.625 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:37:52.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:52.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:53.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:54.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:54.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:54.548 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-16T17:37:54.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T17:37:54.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:54.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:55.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:56.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:56.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:56.290 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:37:56.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T17:37:56.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:56.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:57.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:37:58.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:37:58.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:37:58.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T17:37:58.420 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:37:58.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:37:58.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:37:59.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T17:38:00.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T17:38:00.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T17:38:00.332 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-16T17:38:00.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T17:38:00.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T17:38:00.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T17:38:00.766 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1843686301,v1:127.0.0.1:6819/1843686301] [v2:127.0.0.1:6820/1843686301,v1:127.0.0.1:6821/1843686301] exists,up 99129ba7-e8b7-431e-bae5-b79dbfccef61 2024-09-16T17:38:00.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T17:38:00.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T17:38:00.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T17:38:00.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-16T17:38:01.509 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-16T17:38:01.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-16T17:38:01.528 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-16T17:38:01.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-16T17:38:01.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-16T17:38:01.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-16T17:38:02.219 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:38:02.263 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:38:02.305 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:38:02.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-16T17:38:02.466 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:38:02.492 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:38:02.522 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:38:02.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-16T17:38:02.725 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:38:02.754 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:38:02.784 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:38:02.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-16T17:38:03.019 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:38:03.049 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:38:03.079 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:38:03.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-16T17:38:03.236 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:38:03.265 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:38:03.297 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:38:03.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-16T17:38:03.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-16T17:38:03.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:03.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:03.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:03.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:03.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:03.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:03.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:03.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:03.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:03.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:04.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:04.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:04.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:04.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:04.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:04.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:04.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:04.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:04.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:04.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:04.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:04.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:04.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:04.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:04.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:05.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:05.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:05.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:05.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:05.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:05.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:05.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:05.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:05.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:05.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:05.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:05.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:05.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:05.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-16T17:38:05.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-16T17:38:05.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:05.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-16T17:38:05.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:38:05.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:06.137 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836503 2024-09-16T17:38:06.161 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672973 2024-09-16T17:38:06.184 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509449 2024-09-16T17:38:06.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:09.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:09.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:09.370 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:09.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:09.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:09.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-16T17:38:09.523 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:09.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:09.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:09.683 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:38:09.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:09.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:38:09.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:09.856 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:38:09.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-16T17:38:09.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-16T17:38:09.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:09.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:10.069 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:10.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:10.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:10.187 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:10.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:10.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:10.287 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:10.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:10.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:10.396 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:10.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:10.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:10.522 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:10.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:10.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:10.631 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:10.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:10.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:10.756 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:10.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:10.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:10.865 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:10.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:10.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:11.024 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:11.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:11.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:11.133 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:11.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:11.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:11.266 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:11.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:11.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:11.375 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:11.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:11.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:11.476 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:11.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:11.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:11.568 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:11.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:11.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:11.694 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:11.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:11.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:11.814 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:11.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:11.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:11.911 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:11.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:11.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:12.012 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:12.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:12.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:12.146 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:12.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-16T17:38:12.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-16T17:38:12.289 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:12.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-16T17:38:12.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:12.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-16T17:38:12.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-16T17:38:12.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:12.441 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836506 2024-09-16T17:38:12.465 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672976 2024-09-16T17:38:12.489 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509452 2024-09-16T17:38:12.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:15.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:15.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:15.677 INFO:tasks.workunit.client.0.smithi039.stdout:21 2024-09-16T17:38:15.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:15.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:15.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-16T17:38:15.880 INFO:tasks.workunit.client.0.smithi039.stdout:21 2024-09-16T17:38:15.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:15.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:16.044 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:38:16.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:16.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:16.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-16T17:38:16.202 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-16T17:38:16.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:161: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-09-16T17:38:16.345 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:38:16.373 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:38:16.406 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:38:16.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:162: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-09-16T17:38:16.567 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:38:16.595 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:38:16.623 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:38:16.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:163: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-09-16T17:38:16.775 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:38:16.803 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:38:16.833 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:38:16.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:164: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-09-16T17:38:16.987 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:38:17.017 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:38:17.047 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:38:17.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:165: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-16T17:38:17.207 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: {} 2024-09-16T17:38:17.233 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: {} 2024-09-16T17:38:17.260 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: {} 2024-09-16T17:38:17.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:169: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-16T17:38:17.411 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:17.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:170: TEST_trim_max_entries_with_dups: test_log_size 1.0 18 2 2024-09-16T17:38:17.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:17.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-09-16T17:38:17.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=2 2024-09-16T17:38:17.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:17.561 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836509 2024-09-16T17:38:17.585 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672980 2024-09-16T17:38:17.608 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509455 2024-09-16T17:38:17.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:20.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:20.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:20.793 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:38:20.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:20.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:20.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-09-16T17:38:20.951 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:38:20.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:20.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:21.109 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:38:21.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:21.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:21.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 2 2024-09-16T17:38:21.257 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-16T17:38:21.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:171: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-16T17:38:21.447 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:21.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:172: TEST_trim_max_entries_with_dups: test_log_size 1.0 15 6 2024-09-16T17:38:21.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:21.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-09-16T17:38:21.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=6 2024-09-16T17:38:21.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:21.596 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836512 2024-09-16T17:38:21.621 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672982 2024-09-16T17:38:21.645 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509458 2024-09-16T17:38:21.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:24.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:24.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:24.832 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:38:24.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:24.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:24.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-09-16T17:38:24.988 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-16T17:38:24.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:24.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:25.147 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:38:25.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:25.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:25.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 6 2024-09-16T17:38:25.300 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:38:25.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:173: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-16T17:38:25.417 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:25.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:174: TEST_trim_max_entries_with_dups: test_log_size 1.0 12 10 2024-09-16T17:38:25.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:25.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-09-16T17:38:25.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=10 2024-09-16T17:38:25.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:25.564 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836515 2024-09-16T17:38:25.587 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672985 2024-09-16T17:38:25.610 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509461 2024-09-16T17:38:25.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:28.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:28.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:28.794 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:38:28.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:28.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:28.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-09-16T17:38:28.952 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-16T17:38:28.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:28.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:29.108 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:38:29.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:29.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:29.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 10 2024-09-16T17:38:29.265 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-16T17:38:29.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:175: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-16T17:38:29.403 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:29.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:176: TEST_trim_max_entries_with_dups: test_log_size 1.0 9 14 2024-09-16T17:38:29.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:29.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-09-16T17:38:29.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=14 2024-09-16T17:38:29.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:29.550 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836518 2024-09-16T17:38:29.573 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672988 2024-09-16T17:38:29.597 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509463 2024-09-16T17:38:29.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:32.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:32.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:32.781 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:38:32.782 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:32.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:32.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-09-16T17:38:32.937 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-16T17:38:32.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:32.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:33.092 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:38:33.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:33.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:33.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 14 2024-09-16T17:38:33.248 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-16T17:38:33.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:177: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-16T17:38:33.364 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:33.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:178: TEST_trim_max_entries_with_dups: test_log_size 1.0 6 18 2024-09-16T17:38:33.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:33.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-09-16T17:38:33.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-09-16T17:38:33.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:33.516 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836521 2024-09-16T17:38:33.539 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672991 2024-09-16T17:38:33.563 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509466 2024-09-16T17:38:33.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:36.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:36.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:36.750 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:38:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:36.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-09-16T17:38:36.906 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-16T17:38:36.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:36.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:37.065 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:38:37.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:37.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:37.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-09-16T17:38:37.228 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-16T17:38:37.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:179: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-16T17:38:37.366 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:37.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:180: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-09-16T17:38:37.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:37.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-16T17:38:37.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-16T17:38:37.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:37.517 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836523 2024-09-16T17:38:37.541 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672994 2024-09-16T17:38:37.564 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509469 2024-09-16T17:38:37.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:40.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:40.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:40.750 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:38:40.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:40.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:40.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-16T17:38:40.920 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:38:40.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:40.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:41.079 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:41.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:41.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-16T17:38:41.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:41.234 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:41.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:183: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-16T17:38:41.352 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:41.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:184: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-09-16T17:38:41.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:41.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-16T17:38:41.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-16T17:38:41.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:41.502 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836526 2024-09-16T17:38:41.525 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672996 2024-09-16T17:38:41.548 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509472 2024-09-16T17:38:41.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:44.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:44.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:44.736 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:38:44.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:44.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:44.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-16T17:38:44.893 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:38:44.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:44.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:45.043 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:45.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:45.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:45.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-16T17:38:45.189 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:45.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:185: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-16T17:38:45.305 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:186: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-09-16T17:38:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-16T17:38:45.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-16T17:38:45.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:45.450 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836529 2024-09-16T17:38:45.472 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949672999 2024-09-16T17:38:45.496 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509475 2024-09-16T17:38:45.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:48.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:48.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:48.682 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:38:48.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:48.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-16T17:38:48.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:48.840 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:38:48.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:48.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:48.996 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:48.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:48.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:48.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-16T17:38:49.153 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:49.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:187: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-16T17:38:49.299 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:188: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-09-16T17:38:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-16T17:38:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-16T17:38:49.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:49.450 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836532 2024-09-16T17:38:49.473 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949673002 2024-09-16T17:38:49.497 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509477 2024-09-16T17:38:49.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:52.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:52.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:52.682 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:38:52.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:52.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:52.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-16T17:38:52.838 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-16T17:38:52.838 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:52.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:52.993 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:52.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:52.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:52.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-16T17:38:53.150 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:53.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:189: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-16T17:38:53.285 INFO:tasks.workunit.client.0.smithi039.stderr:error removing test>foo: (2) No such file or directory 2024-09-16T17:38:53.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:190: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-09-16T17:38:53.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-16T17:38:53.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-16T17:38:53.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-16T17:38:53.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-16T17:38:53.434 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0: 21474836535 2024-09-16T17:38:53.459 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1: 42949673005 2024-09-16T17:38:53.482 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2: 64424509480 2024-09-16T17:38:53.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-16T17:38:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:56.500 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:56.671 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:38:56.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:56.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-16T17:38:56.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-16T17:38:56.825 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-16T17:38:56.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:56.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:56.980 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:56.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-16T17:38:56.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-16T17:38:56.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-16T17:38:57.142 INFO:tasks.workunit.client.0.smithi039.stdout:20 2024-09-16T17:38:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-16T17:38:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-16T17:38:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T17:38:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-16T17:38:57.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:38:57.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:38:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:38:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:38:57.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:38:57.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:38:57.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:38:57.456 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:38:57.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:38:57.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:38:57.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:38:57.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:38:57.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:38:57.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:38:57.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:38:57.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:38:57.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:38:57.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T17:38:57.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-16T17:38:57.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:38:57.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:38:57.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:38:57.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.487707 2024-09-16T17:38:57.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:38:57.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:38:57.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T17:38:57.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/repro-long-log 0 2024-09-16T17:38:57.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-16T17:38:57.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T17:38:57.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-16T17:38:57.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T17:38:57.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T17:38:57.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T17:38:57.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T17:38:57.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T17:38:57.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T17:38:57.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T17:38:57.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T17:38:57.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T17:38:57.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T17:38:57.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T17:38:57.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T17:38:57.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:38:57.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T17:38:57.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T17:38:57.497 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T17:38:57.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:38:57.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T17:38:57.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-16T17:38:57.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T17:38:57.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T17:38:57.501 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.487707 2024-09-16T17:38:57.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.487707 2024-09-16T17:38:57.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T17:38:57.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T17:38:57.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T17:38:57.504 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T17:38:57.504 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T17:38:57.517 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-09-16T17:38:57.517 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T17:38:57.877 DEBUG:teuthology.parallel:result is None 2024-09-16T17:38:57.878 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T17:38:57.890 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T17:38:57.890 DEBUG:teuthology.orchestra.run.smithi039:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T17:38:57.944 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T17:38:57.944 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T17:38:57.954 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T17:38:57.955 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T17:38:58.024 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T17:38:58.032 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T17:38:58.033 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T17:38:58.059 INFO:teuthology.orchestra.run.smithi039.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T17:38:58.059 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================== 2024-09-16T17:38:58.059 INFO:teuthology.orchestra.run.smithi039.stdout:*hv01.front.sepi 67.205.162.81 3 u 233 256 377 0.102 -0.269 0.064 2024-09-16T17:38:58.060 INFO:teuthology.orchestra.run.smithi039.stdout:+hv02.front.sepi 63.231.80.2 3 u 263 256 377 0.119 -0.877 0.325 2024-09-16T17:38:58.060 INFO:teuthology.orchestra.run.smithi039.stdout:+hv03.front.sepi 74.6.168.72 3 u 246 256 377 0.092 -0.909 0.058 2024-09-16T17:38:58.060 INFO:teuthology.orchestra.run.smithi039.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-16T17:38:58.061 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T17:38:58.069 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T17:38:58.070 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T17:38:58.078 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T17:38:58.087 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T17:38:58.094 INFO:teuthology.task.internal:Duration was 9642.630197 seconds 2024-09-16T17:38:58.095 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T17:38:58.104 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T17:38:58.104 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T17:38:58.150 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T17:38:58.150 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T17:38:58.150 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T17:38:58.203 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T17:38:58.203 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T17:38:58.282 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T17:38:58.282 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T17:38:58.381 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T17:38:58.391 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T17:38:58.391 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T17:38:58.403 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T17:38:58.425 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:38:58.457 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-09-16T17:38:58.471 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T17:38:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T17:38:58.519 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T17:38:58.535 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T17:38:58.537 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906794/remote/smithi039 2024-09-16T17:38:58.538 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T17:38:58.587 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T17:38:58.587 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T17:38:58.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T17:38:58.637 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T17:38:58.637 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T17:38:58.646 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T17:38:58.646 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T17:38:58.675 INFO:teuthology.orchestra.run.smithi039.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 17:38 /home/ubuntu/cephtest 2024-09-16T17:38:58.676 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T17:38:58.686 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T17:38:58.706 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} duration: 9642.630196809769 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T17:38:58.707 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T17:38:58.792 INFO:teuthology.run:pass